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Preface 


The  diagnosis  and  repair  of  malfunctions  are  of  continuing  importance 
in  the  fields  of  equipment  maintenance.  Recent  advances  in  this  field 
allow  a large  number  of  factors  to  be  measured  more  precisely  than  ever 
before.  If  the  increased  information  resulting  from  this  trend  is  to 
yeild  effective  actions,  advances  of  a similar  nature  are  required  in 
decision  making  as  well. 

The  papers  presented  at  this  symposium  focus  on  the  analysis  of 
decision  making  in  an  environment  where  the  diagnosis  and  repair  of  com- 
plex mechanisms  requires  the  use  of  still  imperfect  measurements  of 
several  factors.  These  analyses  take  several  forms,  among  these  are: 

(1)  estimating  the  effects  of  incorrect  decisions  on  costs,  reliability 
and  other  elements  of  the  loss  function,  (2)  determining  the  variability 
of  tests  and  test  procedures,  (3)  determining  the  best  sequence  of  tests, 
and  (4)  simulating  the  effects  of  alternative  decision  criteria  on 
equipment  repair  processes.  None  of  the  papers  offers  a complete  approach 
to  decision  making  in  this  complex  environment.  However,  each  presents  an 
approach  to  a significant  portion  of  the  problem.  Our  hope  is  that  in 
discussing  and  comparing  the  approaches  presented  here,  our  understanding 
of  the  problem  will  be  increased. 


Norman  Keith  Womer 
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ABSTRACT 

This  report  provides  the  underlying  theory  and  methods  used  to 
determine  the  effect  of  the  Work  in  Process  Inventory  Control  System 
(WIPICS)  on  the  F-4B  to  N conversion  program  and  the  Naval  Air  Rework 
Facility,  North  Island,  California.  This  report  documents  cost  savings 
of  3.24  percent  of  the  "before"  WIPICS  costs.  It  also  concludes  that 
these  cost  savings  are  statistically  significant  at  the  10  percent 
level . 
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THE  EFFECT  OF  WIPICS  ON  THE  F4-B  TO  N 
CONVERSION  PROGRAM 

Section  1 
INTRODUCTION 

The  report  provides  statistical  documentation  of  the  affect  of  the 
Work  in  Process  Inventory  Control  System  (WIPICS)  on  the  Navy's  F4-B  to 
N conversion  program.  This  program  accomplishes  a major  life  extension 
of  the  F4  at  the  Naval  Air  Rework  Facility,  North  Island,  California 
(NARFNI).  The  report  draws  heavily  on  previous  work  in  evaluating  the 
affect  of  complex  systems  on  diverse  programs  especially  the  body  of 
work  on  "Auditing  Cost  Effectiveness  Analyses  of  Technological  Changes" 
accomplished  at  the  U.  S.  Naval  Postgraduate  School  (see  references 
[1,  3,  4,  5,  9,  11,  12]). 

The  purpose  of  this  brief  introductory  section  is  to  explain  the 
philosophy  of  using  estimated  production  and  cost  functions  to  document 
the  benefits  of  a complex  system  like  WIPICS. 

Usually  the  major  source  of  effectiveness  for  an  information  system 
is  factor  saving.  That  is,  an  analysis  will  frequently  document  expected 
effectiveness  in  terms  of  man-days  saved  or  decreases  in  wasted  raw 
materials  or  replacement  of  several  more  expensive  pieces  of  equipment 
or  procedures. 


Other  contributions  to  effectiveness  are  more  difficult  to  estimate 
and  evaluate.  However,  some  attempt  is  usually  made  to  include  effects 
like  increased  speed  of  production,  higher  quality  of  output,  and  more 
control  over  the  production  process. 

Frequently,  an  attempt  is  made  to  assign  dollar  values  to  each  of 
these  measures  of  effectiveness  and  a cost-effective  change  is  defined 
as  one  with  a greater  value  of  effectiveness  than  its  costs. 

Frequently  unstated  in  the  analysis  is  the  assumption  that  these 
effects  are  expected  only  if  nothing  else  changes. 

After  the  system  has  been  installed  one  can  gather  data  cn  the 
changes  that  have  taken  place.  For  example,  we  know  what  has  happened  to 
factor  usage,  we  can  measure  the  new  speed  of  the  production  pr)cess,  and 
perhaps  we  can  determine  quality  changes  in  output.  Here  the  i nportant 
questions  are  not  what  changes  have  taken  place,  but  why  the  changes 
have  occurred.  In  particular  are  the  changes  due  to  the  system? 

Notice  that  after  the  fact,  many  other  things  may  have  changed  as 
well.  In  particular,  the  outputs  of  the  organization  may  have  changed 
in  both  quantity  and  type,  prices  of  factors,  many  have  changed  or  new 
constraints  may  have  been  placed  on  the  organization  or  old  ones  relaxed. 
Finally,  the  manager  of  the  enterprise  may  not  have  acted  exactly  as  the 
cost-effectiveness  analysis  expected.  For  example,  suppose  the  system 
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replaces  old  equipment  and  several  men,  instead  of  laying  off  the  men 
the  manager  reassigns  them  as  trouble  shooters.  As  a result,  output 
increases  dramatically  with  no  increase  in  costs.  Here  costs  did  not 
decline  as  expected,  instead  the  manager  chose  to  increase  output. 

Current  methods  of  auditing  cost-effectiveness  analyses  involve 
looking  at  factors  by  categories  and  determining  if  their  usage  has 
changed.  A portion  of  the  change  in  factor  usage  is  directly  related 
to  the  system  by  verbal  argument  and  a dollar  value  is  assigned  to  that 
portion  of  the  change  in  factor  usage.  Thus  the  relation  between  the 
factor  savings  and  the  system  frequently  is  a verbal  argument  constructed 
by  an  outside  observer.  These  "allowed"  effects  are  then  compared  to 
costs.  Other  effects  like  quality  changes  are  handled  separately. 

Our  method  of  evaluating  the  affect  of  systems  like  WIPICS  recognizes 
that  an  outside  observer  cannot  possibly  trace  second  order  effects  of 
many  decisions  through  a massive  enterprise.  Thus  we  look  at  aggregated 
summary  measures  of  the  organization's  behavior  in  each  of  several  areas 
before  and  after  the  technological  change. 

In  particular  our  approach  to  determining  the  effectiveness  of  WIPICS 
is  to  develop  a model  of  the  production  behavior  of  NARFNI  both  before  and 
after  the  implementation  of  the  system.  These  production  models  are  then 
compared  for  several  sets  of  circumstances  and  conclusions  are  drawn  from 
the  comparisons. 
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This  method  has  the  advantage  of  allowing  the  analyst  to  ask  many 
potential  ‘what  if"  questions  about  the  external  circumstances  affecting 
the  NARF.  It  permits  the  use  of  well  defined  statistical  tests  and 
allows  the  use  of  generally  recognized  economic  theory  in  the  formulation 
of  those  tests.  Finally,  it  is  not  nearly  so  subject  to  the  spurious 
verbal  description  which  frequently  characterize  some  other  methods  of 
evaluation. 
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Section  2 


METHODOLOGY 


The  purpose  of  this  section  is  to  describe  the  assumptions,  limita- 
tions, and  underlying  production  theory  involved  with  determining  a cost 
function  to  describe  the  F4-B  to  F4-N  conversion  program  at  NARFNI. 

The  basic  assumption  is  that  this  production  process  is  adequately 
described  by  a production  function  of  the  Cobb-Douglas^  form.  That  is, 
for  a given  set  of  circumstances: 

X = A La  Me  Dy  (2.1) 

In  (2.1)  X is  a measure  of  the  quantity  of  output  produced. 

L,  M,  and  D are  measures  of  resources  consumed  identified  respectively 
as  direct  man-hours  expended,  material  cost  in  constant  dollars,  and 
number  of  aircraft  days  used.  Production  theory  requires  certain 
restrictions  on  the  parameters  to  be  estimated  in  (2.1).  In  particular 
A > 0 and  05a,  e,  y - 1.  The  above  production  function  implies 
that  increases  in  any  input  permits  an  increase  in  the  output  of  the 
process.  The  production  function  also  permits  decreasing,  constant,  or 
increasing  returns  to  scale  as  a + 6 + y is  less  than,  equal  to,  or 
greater  than  one.  This  formulation  also  implies  that  production  in 

^Sometimes  referred  to  as  the  "Generalized  Cobb-Douglas  Production 
Function." 
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the  Beeline  program  is  independent  of  the  level  of  production  in  the 
other  programs  at  the  NARF.  The  latter  assumption  is  relaxed  to  a 
limited  extent  below. 


The  second  major  assumption  in  the  analysis  is  that  the  NARF 
has  control  only  over  the  quantity  of  inputs  it  uses  and  not  over 
their  unit  costs. 


The  total  cost  of  production  is  defined  as: 

TC  = C0  + DL$  + M$  + P$  (2.2) 

where  C0  is  overhead  cost  including  production  expense  applied  and 
general  and  administrative  expense,  DL$  is  direct  labor  cost,  M$  is 
direct  material  cost,  and  P$  is  a penalty  cost  for  the  imputed  value 
of  an  aircraft  day  in  shop. 

The  assumption  of  exogenous  unit  factor  costs  implies  that: 

DL$  = PrL  (2.3) 

where  P is  exogenous; 

M$  = Pm-M  (2.4) 

where  Pm  is  exogenous;  and 
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where  P^  is  exogenous. 


Like  all  assumptions  these  are  not  expected  to  be  fulfilled  exactly 
but  there  is  some  reason  to  expect  that  to  a large  extent  the  NARF  does 
not  exert  control  over  these  variables.  Clearly  Civil  Service  wage 
scales  are  not  under  the  control  of  the  NARF.  In  addition  the  NARF's 
ability  to  adjust  the  experience  level  of  its  personnel  to  alter  the 
average  wage  rate  is,  at  best,  a long  run  phenomenon.  In  any  case  this 
fluctuation  in  average  wage  rate  is  more  usually  a result  of  Navy  wide 
policy  than  of  internal  response  to  production  circumstances.  This  is 
not  to  say  that  the  NARF  is  totally  unable  to  influence  average  wage 
rates,  but  merely  to  point  out  that  the  major  sources  of  fluctuation 
in  this  variable  are  outside  the  control  of  the  NARF. 

The  other  two  variables  that  are  assumed  to  be  exogenous  are  more 
clear  cut.  Pm  is  a price  index  for  material  costs  that  is  determined 
by  economic  conditions  and  the  policies  of  the  Naval  Supply  system. 

Pd  is  an  imputed  value  of  an  aircraft  day  in  shop  and  is  determined 
by  the  usefulness  of  a day's  service  by  an  F4-N.  This  variable  is 
also  not  under  the  control  of  the  NARF. 


Another  assumption  is  used  to  derive  a cost  function  for  the  Beeline 
program,  ihe  NAKF  is  presumed  to  minimize  the  cost  of  any  given  level  of 
output  subject  to  the  constraint  imposed  by  the  production  function. 
Formally  the  assumption  is  that  the  NARF  chooses  values  of  L,  M.  and  D 
to  solve  the  program: 


Min  TC  = C0  + PrL  + Pm-M  + Pd*D 
s.t.  X = A La  M6  Dy 


(2.6) 


The  solution  to  this  program  yields  total  cost  as  a function  of 
output  and  the  exogenous  variables  as  defined  below. 


TC  = C + A*  X1/ct+6+Y  pa/a+S+Y  pB/a+B+y  py/a+B+y 
0 2.  m d 


(2.7) 


This  is  the  basic  form  of  the  cost  function  for  the  Beeline  pro- 
gram corresponding  to  a given  set  of  circumstances. 

The  use  of  (2.7)  as  a reduced  form  cost  function  involves  one 
further  assumption.^  That  is  that  the  level  of  output,  X,  is  also 
an  exogenous  variable  to  the  system,  specified  by  the  fleet  or  at 
least  determined  prior  to  beginning  work  on  the  project.  This  last 
assumption  is  subject  to  some  criticism  for  some  measures  of  output. 


^See  Nerlove  [ 8 ] for  a more  formal  elaboration  of  these  assumptions 
and  the  role  they  play  in  model  construction.  The  comments  by  Dhrymes 
[2  , p.  234]  on  the  requirement  for  imperfect  cost  minimization  are 

also  appropriate. 
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In  particular  in  an  environment  where  production  load  norm  is  renegoti- 
ated during  or  after  the  completion  of  a job;  it  would  not  be  a measure 
of  output  which  satisfied  this  assumption.  Alternative  measures  of 
output  are  suggested  below. 

Changing  Ci rcumstances  of  Production 

Up  to  this  point  the  derivation  of  the  cost  function  has  presumed 
a given  set  of  circumstances.  Two  circumstances  which  may  be  expected 
to  change  over  time  are  the  efficiency  of  labor  due  to  a learning  effect 
and  the  efficiency  of  production  on  the  Beeline  program  due  to  inter- 
action with  other  F4  programs  at  the  NARF  which  use  some  of  the  same 
resources. 

To  handle  the  first  of  these  changing  circumstances  suppose  that 
L,  the  labor  referred  to  in  equation  (2.1),  represents  labor  that 
is  as  efficient  as  that  used  to  produce  the  first  B to  N conversion. 
If  labor  is  subject  to  a learning  effect  of  "a"  then  labor  in  the 
first  conversion  may  be  related  to  labor  in  the  n th  conversion  by 

Ln  = Na/*n2  L (2.8) 

where  L represents  the  labor  required  in  the  first  conversion. 
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Rewriting  (8)  we  have 


where 


1 


L = Ln  N 


6 = - a/in  2 


(2.9) 


If  (2.9)  is  substituted  into  (2.1)  the  production  function  is 
now  defined  in  terms  of  labor  of  appropriate  efficiency  for  the  n th 
conversion. 


Following  through  the  analysis  above  the  cost  function  may  be  changed 
to  reflect  the  learning  effect  as: 


TC  = C0  + A'  XVa+6+Y  p“/a+6+T  pS/a+e+Y  pY/a+6+y  N-a<5/a+B+Y  ^ 


10) 


The  interaction  between  production  on  the  Beeline  program  and  other 
F4  programs  at  the  NARF  can  be  handled  in  a similar  fashion.  Suppose 
A in  equation  (2.1)  represents  the  efficiency  of  production  in  the 
Beeline  program.  Further  suppose,  that  A depends  on  K,  the  number 
of  F4's  in  the  953  shop,  in  the  following  way. 


A = B Kw 


(2.11) 


Substituting  from  (2.11)  into  (2.1)  and  following  the  analysis 
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a,  the  natural  log  of  the  learning  curve  slope  is  expected  to  be 
less  than  0,  therefore  6 > 0,  i.e.  £n  .8  = - 0.223  for  an  eighty 
percent  learning  curve. 
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above  the  cost  function  may  be  written  as: 


TC  = C0  + B1  X1/a+8+T  pa/a+B+Y  pB/ct+B+y  pYj/a+B+Y  ^-afi/a+B+y  «-u)/a+8+Y 

(2.12) 

Equation  (2.12)  may  be  estimated  in  either  the  before  WIPICS 
or  the  after  WIPICS  environments  by  the  following  transformation: 


log(TC-C0)  = 60  + Bi 1 og  X + 62  log  P£  + 83  log  Pm  + 84  log  N + 65  log  K + e 

(2.13) 

where1  S0  = log  B'  + Pd 

1_ 

81  a+S+y 

8 2 a+8+y 

R = B 
M3  a+8+y 

p.  = ~a5 

"4  a+8+y 


Be  = — 1 '±— 

0 a+8+y 

To  be  consistent  with  the  assumptions  above  the  following  restrictions 
on  the  values  of  the  estimated  parameters  must  be  satisfied: 


^ince  all  observations  are  on  F4-N's,  the  penalty  cost  per  aircraft 
day  does  not  change  from  observation  to  observation  and  P<j  is  a 
constant. 
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o < 60  , 1/3  < B1  , 0 < B2  < 1 , 

0-63-1  , and  - 1 < g4  5 0. 

The  last  restriction  allows  for  learning  curves  ranging  from 
fifty  to  one-hundred  percent. 
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DESCRIPTION  OF  AVAILABLE  DATA 


The  preceding  mathematical  model  is  defined  in  terms  of  physical 
measures  of  inputs  and  outputs  and  well  defined  prices  for  these  quanti- 
ties. In  fact,  data  on  these  variables  cannot  be  obtained  in  most  large 
organizations  and  NARFNI  is  no  exception.  The  data  obtained  from  NARFNI 
consists  of  information  on  83  F4-B  to  N conversions  begun  during  the 
period  July  1972  to  December  1973.  For  each  conversion  the  following 
information  was  available: 

(1)  Identification  number 

(2)  Induction  data 

(3)  Production  date 

(4)  Production  load  norm 

(5)  Direct  labor  hours  expended 

(6)  Direct  labor  cost 

(7)  Direct  material  cost 

(8)  Applied  overhead  cost 

(9)  Applied  statistical  cost 

The  manner  in  which  the  above  statistics  are  accumulated  and  used  by 
NARFNI  are  enumerated  in  [ 6 ].  The  data  is  listed  for  reference 
purposes  in  Appendix  A. 

The  induction  and  production  dates  of  each  F4  rework  job  in 
shop  during  the  period  July  1972  to  December  1973  was  also  provided. 


Also  available  was  data  on  the  standard  hours  produced  and  the  number 
of  man  houi-s  required  to  produce  them  by  the  953-shop  (aircraft  assembly). 
Prior  to  1 January  1973  this  information  was  available  only  on  a quarterly 
basis  but  after  that  time  weekly  information  was  used. 

One  additional  item  of  information  concerns  a penalty  cost  assigned 
to  aircraft  down  time  at  the  NARF.  This  penalty  cost  per  day  was  calcu- 
lated from  average  unit  flyaway  procurement  cost  for  the  F4-J.  The 
penalty  cost  was  calculated  as  follows: 

[Unit  cost  of  F4-J/Service  Life]  = [Penalty  cost/day  in  shop]  (3.1) 

A unit  cost  of  $2,492,000  and  an  expected  service  life  of  2400  days  (80 
months)  were  used  to  obtain  a daily  penalty  cost  of  $1038.23. 

These  data  were  used  to  form  several  proxies  for  the  variables  of 
Section  Two.  A brief  description  of  each  follows: 

1.  Output.  If  it  were  the  case  that  an  F4-B  to  N conversion  required 
exactly  the  same  repairs  and  modifications  for  each  aircraft  there  would 
be  no  problem  with  output  measurement.  The  completion  of  each  job  requires 
bringing  each  aircraft  up  to  a given  level  of  modification  and  an  extensive 
inspect  and  repair  as  necessary  operation.  Different  aircraft  require 
different  modifications;  and  repairs  are  made  on  some  aircraft  that  are 
not  necessary  on  others.  Hence,  the  level  of  effort  expended  on  a con- 
version depends  on  the  particular  aircraft  involved.  Thus  the  output  of 
the  conversion  effort  is  not  the  number  of  F4-N's  produced.  It  is 
rather  a measure  of  the  difference  between  the  F4-B  that  the  fleet  gave 
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up  and  the  F4-N  that  was  returned  to  it.  This  is  the  output  which  we 
attempt  to  measure. 

Clearly  none  of  the  data  available  directly  measure  output  of  the 
conversion  effort,  hence  a proxy  measure  of  some  kind  is  necessary.  One 
possibility  is  to  merely  ignore  differences  in  jobs  and  act  as  if  each 
job  produced  the  same  output.  While  the  results  of  this  assumption  are 
also  reported,  a proxy  variable  based  on  an  index  of  standard  hours  was 
used  as  our  measure  of  output. 

This  index  attempts  to  measure  work  content  of  a given  job  by  the 
class  A,  B,  C,  or  D standard  hours  associated  with  the  job.  These 

standards  tend  to  be  stable  over  time.  Thus  increases  in  the  number  of 

standard  hours  tend  to  indicate  more  extensive  repairs  or  modifications 
for  a job  and  therefore  more  output  produced  upon  completion  of  the  job. 
The  method  for  calculating  the  index  of  standard  hours , S,  is  reported 
in  Appendix  B. 

2.  Labor.  Direct  man-hours  expended,  L,  are  used  as  a measure 
of  labor  used.  This  variable  excludes  indirect  man-hours  and  fails  to 
distinguish  among  alternative  types  of  labor  used.  In  spite  of  these 
drawbacks,  it  is  one  of  the  most  accurate  of  the  proxy  variables. 

3.  Wage  rate.  The  ratio  of  direct  labor  costs,  L$,  to  direct 

man-hours  expended,  L,  is  used  as  a proxy  for  wage  rate  P^.  Because 

this  variable  is  averaged  over  all  types  of  labor  it  may  be  influenced 
by  factors  other  than  a general  increase  in  wages.  For  example  a 
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modification  that  requires  very  technical  work  and  uses  high  priced 
labor  exclusively  will  be  associated  with  higher  values  of  this  proxy. 
Also,  periods  of  time  that  involve  large  quantities  of  overtime  labor 
will  show  high  values  of  the  variable.  Still,  this  variable  is  thought 
to  be  a reasonably  accurate  measure  of  the  wage  rate. 

4.  Material.  No  attempt  has  been  made  to  construct  a physical 
measure  of  the  quantity  of  material  consumed  on  each  job.  Instead  the 
cost  of  material  used  at  FY  1973  prices  was  used  as  a proxy  for  the 
quantity  of  material. 

5.  The  Price  of  Material.  An  index  of  the  material  price  change 

experienced  1 July  1973  was  formed  as  follows:  The  value  of  all  identi- 

fiable material  expended  on  the  F4-B  to  N conversion  program  during  the 
fourth  quarter  of  FY  73  was  compared  to  the  cost  of  that  material  at 
prices  in  effect  31  December  1974.  This  difference  $453,752.92  - 
$438,333.58  was  used  to  calculate  the  percentage  increase  in  material 
prices  as  3.52%.  Thus  the  material  price  index,  Pm,  our  proxy  for 
the  price  of  material  is  1.00  for  periods  prior  to  1 July  1973  and 
1.0352  after  that  time. 

6.  Number  of  days  in  shop.  It  seems  clear  that  if  output  is  to 
be  a measure  of  modifications  and  repairs  to  F4-B 1 s then  one  input  must 
be  the  number  of  aircraft  days  in  shop.  This  variable  is  denoted  D. 


17 


7.  Penalty  cost  per  aircraft  day.  The  NARF  does  not  pay  for 
F4-B's  to  convert  nor  does  it  provide  "loaners"  to  the  fleet  while  the 
conversion  is  taking  place.  Nevertheless,  from  the  point  of  view  of 
the  Navy,  the  pipeline  of  F4-N's  in  the  conversion  process  is  costly 
and  both  the  NARF  and  the  Navy  are  concerned  with  reducing  the  size  of 
the  pipeline.  Hence  our  study  includes  as  costs  of  the  NARF  a penalty 
cost  per  aircraft  day  of  $1038.23.  As  explained  above  this  penalty 
cost,  Pq,  is  based  on  the  procurement  cost  of  an  F4-J;  the  assumption 
being  that  those  aircraft  replacing  F4-B 1 s in  the  fleet  were  F4-J's 
not  B's.  Therefore  the  appropriate  opportunity  cost  is  related  to  the 
procurement  of  F4-J’s. 

8.  Number  of  F4 1 s in  shop.  In  the  attempt  to  capture  any  inter- 
actions between  the  Beeline  and  other  programs  at  the  NARF  the  number 
of  F4's  in  the  953  shop  at  any  point  in  time  was  measured.  The  number 
of  F4's  in  shop  averaged  over  the  days  that  any  given  Beeline  job  was 

in  shop  was  included  as  a variable  in  the  regression  analysis  to  accou’nt 
for  this  interaction. 

9.  Sequence  number.  In  an  attempt  to  correct  for  learning  that 
may  have  been  taking  place  as  the  Beeline  program  progressed  the  sequence 
number  of  each  job  was  also  recorded  and  used  in  the  regression  analysis. 

These  aggregated  data  were  used  as  described  in  Section  4 to 
estimate  the  parameters  of  the  cost  models. 
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Section  4 


ESTIMATION  AND  HYPOTHESIS  TESTS 
A.  Preliminary  Data  Analysis 

The  nature  of  the  available  data  described  in  the  previous  section 
dictated  certain  regression  techniques  in  the  estimation  of  the  before 
and  after  cost  function. 

While  data  was  provided  on  83  jobs  only  52  of  these  were  completed. 
The  other  31  jobs  were  in  various  stages  of  completion.  The  data  pro- 
vided for  these  later  jobs  contained  estimated  costs  and  time  to  comple- 
tion and  as  a result  it  was  decided  not  to  make  use  of  this  information. 
In  addition  one  of  the  52  jobs  that  was  completed  had  been  inducted  with 
crash  damage.  It  was  felt  that  the  nature  of  the  work  involved  on  that 
job  was  sufficiently  different  from  the  others  to  justify  dropping  it 
from  the  data  base.  Thus  51  jobs  were  available  for  the  estimation. 

Of  the  51  jobs  25  were  still  inprocess  after  1 July  1973,  the  date 
the  chaining  feature  of  WIPICS  first  became  available  for  use.  However, 
no  data  was  available  on  completed  jobs  inducted  after  1 July.  As  a 
result  it  was  decided  to  estimate  a single  cost  function  for  Beeline 
jobs  with  a dummy  variable  for  WIPICS  instead  of  two  separate  cost 
functions  one  before  and  one  after  1 July. 
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This  procedure  requires  a specification  of  how  WIPICS  affects  the 
cost  function  (in  this  case  by  a one  time  shift)  and  therefore  is  less 


able  to  account  for  more  subtle  affects.  On  the  other  hand  this 
approach  avoids  the  allocation  of  observations  to  two  functions  and 

the  autocorrelation  introduced  by  prorating  jobs  over  time  required 

by  the  alternative.  It  also  avoids  the  hypothesis  testing  problem  in 
environments  where  the  sample  variance  may  differ  between  the  before 
and  after  situation. 

The  dummy  variable  was  defined  as  the  proportion  of  time  a job  was 
in  shop  before  1 July.  A second  variable  (one  minus  the  first  one) 
expresses  the  proportion  of  time  a job  was  in  shop  after  1 July.  By 
this  device  equation  (2.13)  was  transformed  to: 

log  (TC-C0)  = Bjj  W + Ba  ( 1 — W ) + B-j  log  X 

+ B2  log  P^  + B3  log  Pm  + B4  log  N 

+ &5  log  K + e (4.1) 


where  W 
W 


1 if  production  date  is  before  1 July  and 
(days  in  shop  before  1 July)/days  in  shop 
if  production  date  is  after  1 July. 


20 


y 


This  does  not  yield  a linear  effect  of  time  after  1 July  on  costs 
because  ot  the  convex  transformation  involved.  This  has  the  effect  of 
giving  jobs  only  slightly  after  1 July  slightly  more  emphasis  than  they 
deserve  thus  we  penalize  the  system  slightly.  However  these  considera- 
tions are  expected  to  be  minor.  The  approach  also  assumes  that  the 
effect  of  WIPICS  on  cost  savings  of  jobs  is  evenly  distributed  over  the 
time  a job  is  in  shop. 

The  data  also  created  one  additional  change.  The  price  index  for 
material  experienced  only  one  change,  on  July  1,  1973  the  estimated 
prices  of  material  increased  3.52%  as  computed  by  NARFNI  [7],  Thus 
the  material  price  index  is  perfectly  col  linear  with  the  dummy  variable 
due  to  WIPICS  and  coefficients  for  neither  are  able  to  be  estimated 
alone.  Thus  the  cost  function  contains  a dummy  variable  for  effects 
due  to  discreet  changes  occurring  on  1 July  1973  and  not  just  due  to 
WIPICS.  A method  recommended  by  Theil  [10]  as  explained  in  Appendix  C 
is  used  to  separate  these  two  affects  and  the  price  index  for  material 
is  dropped  from  the  cost  function. 

Prior  to  estimating  the  cost  function  the  matrix  of  estimated  correla- 
tion coefficients  among  the  proposed  independent  variables  was  examined. 

The  lower  triangle  of  the  matrix  is  recorded  in  Table  1.  From  an  examin- 
ation of  the  table  it  seems  clear  that  several  of  the  variables  are 
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highly  correlated  with  one  another.  That  is  log  X,  log  P^,  and  log  X 
seem  to  contain  approximately  the  same  information.  This  result  is  quite 
obvious  in  the  regression  results  reported  below.  It  also  seems  clear 
that  the  dummy  variable,  W,  is  only  slightly  correlated  with  the  other 
variables. 


TABLE  1 

SIMPLE  CORRELATION  COEFFICIENTS 


w 

log  Pa 

log  N 

log  K 

log  PA 

-0.175 

log  N 

-0.691 

0.497 

log  K 

-0.055 

0.991 

0.402 

log  X 

-0.116 

0.987 

0.385 

0.991 

B.  Estimating  the  Cost  Function 

In  the  process  of  estimating  the  cost  function  one  fact  became 
painfully  obvious.  No  matter  which  variables  were  included  in  the 
regression  the  first  observation  was  an  outlier.  In  all  cases  its 
residual  error  was  more  than  three  times  the  standard  deviation  indi- 
cating an  extremely  low  probability  of  that  observation  being  a member 
of  the  same  population  as  the  others.  For  these  reasons  the  first 
observation  was  dropped  leaving  a sample  of  size  50. 
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Because  of  the  high  degree  of  multi  col  linearity  attempts  to  include 
all  of  the  variables  in  the  regression  proved  to  be  fruitless.  Several 
regressions  are  reported  below.  Each  contains  a different  set  of  explana- 
tory variables  for  the  log  of  variable  cost,  direct  labor  cost  plus  material 
cost  plus  penalty  cost.  In  all  cases,  the  regression  is  reported  first 
with,  then  without  including  X our  measure  of  output.  The  numbers  in 
parenthesis  below  the  coefficients  are  t statistics  and  S is  the  esti- 
mated standard  deviation  of  the  regression. 

log  (VC)  = 2.94  W + 2.92  (1-W)  + 0.20  log  Pz  + 0.61  log  X (4.2) 

(5.29)  (5.18)  (0.50)  (9.07) 

S = .022 

Here  the  coefficient  of  the  wage  rate  is  clearly  not  significant  while  the 
other  coefficients  are  each  significant  at  the  5 % level. 

log  (VC)  = 7.21  W + 7.24  (1-W)  - 1.75  log  P*  (4.3) 

(14.81)  (14.46)  (-3.01) 

S = .036 

If  X is  dropped  from  the  relation  the  coefficient  of  log  P^ 
changes  dramatically  indicating  multicollinearity.  Also  the  standard 
deviation  increases  by  over  50  percent.  It  seems  clear  that  we  must 
choose  between  log  X and  log  P^  and  the  choice  is  clearly  to  include 
log  X. 
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log  (VC)  = 3.00  W + 2.97  (1-W)  + 0.01  log  N + 0.61  log  X 


(4.4) 


(9.14)  (8.76)  (0.86)  (8.64) 

S = .022 

Here  the  coefficient  of  the  sequence  number  is  clearly  not  signifi- 
cant while  the  other  coefficients  are  significant  at  the  5 percent  level. 

log  (VC)  = 5.83  W + 5.89  (1-W)  - 0.078  log  N (4.5) 

(235.92)  (127.85)  (-3.65) 

S = .035 

If  X is  dropped  from  the  relation  we  do  get  a negative  coefficient 
for  log  N as  we  would  expect  but  the  standard  deviation  increases  by 
over  50  percent.  It  seems  clear  that  relation  (4.4)  is  preferable  to 
(4.5)  and  likewise  that  the  contribution  of  sequence  number  to  explaining 
cost  is  nil. 

log  (VC)  = 1.71  W + 1.73  (1-W)  + 0.66  log  K + 0.67  log  X (4.6) 

(2.01)  (2.08)  (1.82)  (9.69) 

S = .021 

Here  the  coefficient  of  number  of  F4  jobs  in  shop  is  not  quite  signifi- 
cant at  the  5 percent  level  while  the  other  coefficients  are. 
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I _•*.  idl 


log  (VC)  = 8.23  W + 8.14  (1-W)  - 1.42  log  K 


(4.7) 


V9.24)  (5.43)  (-2.80) 

S = .036 

While  the  results  of  (4.6)  are  not  by  themselves  disturbing  the 
comparison  of  (4.7)  to  (4.6)  indicates  a rather  dramatic  change  in  the 
coefficient  of  log  K when  log  X is  removed  from  the  equation.  Thus 
indicating  multicollineari ty.  It  seems  clear  that  log  X is  preferable 
to  log  K since  the  standard  deviation  of  (4.7)  is  more  than  50  percent 
higher  than  that  of  (4.6). 

The  previous  regression  results  lead  to  a formulation  which  because 
of  multi  coll inearity  and  the  lack  of  significance  of  the  sequence  number 
includes  only  the  log  of  output  in  addition  to  the  dummy  variables  for 
WIPICS  in  the  cost  function. 

log  (VC)  = 3.186  W + 3.174  (1-W)  + 0.595  log  X (4.8) 

(12.99)  (12.92)  (10.42) 

S = .0219 

Here  all  the  variables  are  easily  significant  at  the  5 percent  level 
and  despite  the  fact  that  only  three  explanatory  variables  are  used  the 
estimated  standard  deviation  is  low.  Hence  (4.8)  is  at  least  as  good 
as  the  other  relations  investigated  in  terms  of  goodness  of  fit,  it  is 
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the  simplest  of  the  relations  and  the  sign  and  magnitude  of  all  the 
coefficients  are  believable  and  consistent  with  the  theoretical  consider- 
ations which  underlie  the  cost  function.  Thus  (4.8)  is  the  preferred 
regression  equation. 

2 1 2 

R for  the  last  regression  is  .70  and  R adjusted  for  degrees 

of  freedom  is  .68.  The  residuals  of  (4.8)  were  examined  for  auto- 
correlation using  the  Durbin-Watson  statistic  and  the  null  hypothesis 
of  serial  independence  failed  to  be  rejected  at  the  95  percent  level. 

C.  Hypothesis  Tests 

It  is  a straightforward  matter  to  test  the  hypothesis  that  expected 
variable  costs  in  the  period  after  1 July  1973  are  lower  than  before 
1 July  other  things  remaining  the  same.  This  is  just  a one  tailed  test 
that  a particular  linear  combination  of  the  parameters  of  (4.8)  are 
less  than  or  equal  to  zero. 

That  is: 

a = E (log  (VCb)  - log  (VCa))  = 8b  - Ba  (4.9) 

From  (4.8)  a may  be  estimated  as  3.186  - 3.174  = 0.0124.  The 
standard  deviation  of  a,  Sa  may  also  be  estimated  from  the  variance- 
covariance  matrix  associated  with  (4.8).  Here  Sa  = .0110.  Thus  a t 
statistic  may  be  calculated  as 
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t = a/Sa  = 1.125 

This  statistic  indicated  that  the  hypothesis  that  "before"  costs  are  less 
than  or  equal  to  "after"  costs  must  be  rejected  in  favor  of  the  alterna- 
tive hypothesis  at  the  15  percent  level. 

This  result  can  be  strengthened  by  removing  the  affect  of  the  material 
price  increase  from  the  dummy  variables  in  (4.8).  The  procedure  used  is 
based  on  Theil  [10,  pp.  147-155]  and  is  reported  in  Appendix  C.  These 
results  from  the  basis  for  Section  5. 
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Section  5 


RESULTS  AND  CONCLUSIONS 


The  probability  that  there  were  cost  savings  due^  to  WIPICS  sig- 
nificantly greater  than  $0  is  greater  than  90%.  The  estimated  cost 
savings  due  to  WIPICS  are  3.24%  of  the  "before"  WIPICS  costs. 


These  conclusions  are  reached  by  the  regression  analysis  reported 
in  Section  4 which  determines  a cost  function  for  the  Beeline  program 
in  each  of  two  environments;  one  before  the  chaining  feature  of  WIPICS 
became  available  for  use  on  the  program,  and  one  after  the  feature  was 
usable. 


The  "before"  cost  function  is  estimated  to  be: 
log  Cg  = 3.186  + .595  log  X 
(12.99)  (10.42) 


The  "after"  cost  function  is  estimated  to  be:2 
log  CA  = 3.174  + .595  log  X 
(12.92)  (10.42) 


Hhat  is,  the  cost  savings  were  associated  with  an  event  which  coincided 
with  the  introduction  of  the  chaining  feature  of  WIPICS  to  the  F4-B  to  N 
conversion  program.  Furthermore  this  effect  was  in  addition  to  effects 
associated  with  changes  in  workload,  wage  rates,  material  prices,  the 
number  of  F4  jobs  in  shop,  and  learning  as  expressed  by  sequence  number. 

2The  "two"  functions  are  estimated  simultaneously  with  the  specification 
that  they  differ  from  one  another  only  by  the  constant  term.  The  co- 
efficient of  determination,  r2,  is  .70  and  the  estimated  standard  devia- 
tion, S,  is  .0219.  These  results  apply  to  both  functions.  The  numbers 
in  parentheses  are  t statistics. 
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In  these  relations  C A and  Cg  represent  variable  costs  of  the 
Beeline  program  in  the  before  and  after  environments.  The  variable 
costs  include  direct  labor  cost,  material  cost,  and  a penalty  cost 
for  number  of  days  in  shop. 

From  the  estimated  cost  function  one  can  test  the  hypothesis 
that  variable  costs  in  the  after  period  are  significantly  lower  than 
in  the  before  period.  To  do  this  we  estimate  the  log  of  the  ratio  of 
"before"  costs  to  "after"  costs,  a. 

That  is: 

a = log  (CB/CA)  = log  CB  - log  CA 
= 3.186  - 3.174  = 0.0124 


The  standard  deviation  of  a,  Sfl  is  estimated  as: 
Sa  = .0110 


Therefore  a t statistic  may  be  calculated  as: 


+ — a _ 0.0124  _ i ioc 

t • ' om  ' 1J25 


This  statistic  indicates  that  the  hypothesis  that  "before"  costs 
are  not  greater  than  "after"  costs  must  be  rejected  at  the  15%  level. 
That  is,  if  it  were  true  that  there  are  no  improvements  in  costs  in 
the  "after"  situation,  the  probability  of  obtaining  such  a large  value 
as  1.125  is  less  than  15%. 
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The  estimated  value  of  a at  0.0124  corresponds  to  an  estimated 
decrease  in  costs  of  2.82%. 

The  empirical  distribution  of  the  percentage  cost  decrease  is 
indicated  in  figure  1. 

These  results  indicating  a significant  cost  savings  of  2.82%  in 
the  "after"  period  understate  the  case  concerning  the  influence  of 
the  WIPICS  chaining  features  on  costs.  This  is  because  a major  influ- 
ence on  costs  in  the  "after"  period,  the  affect  of  a 3.52%  in  average 
material  prices,  has  not  been  separated  from  the  affect  of  WIPICS  on 
costs. 

The  following  sensitivity  analysis  may  be  used  to  unravel  these 
two  entwined  affects.  The  fact  that  the  dummy  variable  associated  with 
WIPICS  is  perfectly  col  linear  with  the  price  index  for  material  requires 
that: 


3.186  = B + D log  PB 

That  is,  the  estimated  constant  term  in  the  "before"  cost  function  is 
not  just  an  estimate  of  the  productivity  before  WIPICS.  It  also  includes 
the  influence  of  the  price  of  material  in  the  "before"  situation.  Likewise, 

3.174  = A + D log  P^ 

Where  P^  is  an  index  of  the  price  of  material  after  1 July  1973. 


Figure  1 


Since  we  know  that  material  prices  have  increased  3.52%  since 
1 July  1973^  we  can  with  perfect  generality  assign  values  of  1.0  and 
1.0352  to  and  Pg  respectively.  Thus,  to  correct  the  difference 
between  these  numbers  for  the  influence  of  price  increases  on  material, 

we  need  only  know  the  value  of  D,  the  relation  between  material  prices 
and  variable  costs  of  a Beeline  job.  Unfortunately,  because  of  the 
multi  coll inearity  problem,  there  is  no  way  of  estimating  the  value  of 
D.  However,  it  seems  clear  that  D will,  in  general,  take  on  values 
only  in  the  range  of  0 to  1.  That  is  clearly  D is  a positive  number, 
if  material  prices  increase  and  nothing  else  happens  we  surely  expect 
the  variable  costs  of  a Beeline  job  to  increase  by  some  amount.  Like- 
wise, it  seems  clear  that  if  material  costs  doubled,  so  that  P^  were 
one  and  Pg  were  two,  we  would  not  expect  the  variable  cost  of  a Beeline 
job  to  also  increase  by  a factor  of  two.  As  a matter  of  fact,  it  can 
be  shown  that  if  the  assumption  of  the  Cobb-Douglas  production  functions 
is  met,  then  the  maximum  value  for  D is  the  coefficient  of  output  in 
the  cost  function,  .59.  Thus  D is  expected  to  lie  in  the  range  0 to  0.59. 


Figure  1 corresponds  to  the  extreme  case  where  D is  assumed  equal 
to  0.  Figure  2,  based  on  the  opposite  extreme,  D = 0.59,  indicates 
that  if  this  were  the  case,  cost  savings  due  to  WIPICS  would  be  estimated 
at  4.78%.  Finally,  a far  more  believable  intermediate  case  is  presented 


^ee  reference  [7]. 
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in  Figure  3.  Here  D is  assumed  to  be  equal  to  the  proportion  of  material 
cost  to  variable  cost  for  a Beeline  job,  approximately  0.125.  In  this 
case,  cost  savings  due  to  WIPICS  are  estimated  at  3.24%. 

This  last  case  with  a 70%  confidence  interval  in  the  range  0.81% 
to  5.67%  and  an  estimated  cost  savings  due  to  WIPICS  of  3.24%  that  is 
significantly  different  from  zero  at  the  10%  level  is  thought  to  be 
the  best  representation  of  the  productivity  increase  documented  by  a 
cost  savings  due  to  WIPICS. 
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APPENDIX  A 

F4-B  to  N Conversion  Data 


K ID  PD 


L$  M$  PE 


GA  NC  SC  TC 


1 2183  2355  34346  34403  225876  59087  99674  141676  526313  220010  746323 

2 2187  3081  29500  38112  251785  51750  110221  160242  573998  228428  802426 

3 2196  3075  41672  41717  274861  58879  123167  176153  633060  226492  859552 

4 2215  3081  36252  41472  275288  84975  125035  177124  662422  283548  945970 

5 2223  3089  39177  39666  263031  60807  113793  173078  610709  227066  837775 

6 2231  3117  40462  40462  271404  85495  114126  179411  650436  1 98382  848818 

7 2238  3117  40783  40783  275167  73879  117683  182711  649440  230452  879892 

8 2245  3134  40220  40220  271970  83419  114296  181513  651198  274465  925663 

9 2255  3144  40130  40130  273723  95217  117579  181557  668076  278714  946790 

10  2258  3149  29500  38354  278233  62368  113167  176849  630617  208864  839481 

11  2264  3177  42103  42178  290183  72895  125876  195442  684396  266180  950576 

12  2270  3151  31000  32026  218319  60272  93261  148714  520566  221580  742146 

13  2277  3120  29100  31485  213561  59411  92808  143418  509198  110310  619508 

14  2280  3128  34100  33586  229457  66889  99825  154874  551045  135837  686882 

15  2286  3151  29100  36218  251945  57404  109211  168712  587272  151968  739240 

16  2290  3162  28600  31339  214480  58157  94170  146730  513537  128416  641952 

17  2292  3151  28800  36239  249338  59937  107794  170246  587315  137374  724689 

18  2298  3151  27600  31203  217535  55954  92221  147406  513116  159433  672549 

19  2300  3169  26600  28891  202092  53899  83855  137839  477685  108413  586098 

20  2306  3212  40812  42267  290390  59523  123176  203286  676375  155714  832089 

21  2311  3173  28100  30412  213362  54784  92372  146129  506647  226142  732789 

22  2314  3166  27100  31463  223109  48810.  89836  151116  512871  138247  651118 

23  2319  3179  28100  33950  239320  54734  100184  164031  558269  208502  766771 

24  2322  3166  29100  36074  255034  55997  103962  174361  589354  205340  794693 

25  2326  3166  28100  31020  218843  52296  86042  150429  507610  214239  721849 

26  2332  3179  29100  37874  266316  58966  103844  184866  613992  210340  824332 

27  2335  3180  29100  34543  244793  57191  103121  169547  574652  220358  795010 

28  2341  3234  34846  36066  250786  59628  103372  175817  589683  216512  806115 

29  2346  3270  32100  32017  223837  70031  101469  153025  548502  229419  777921 

30  2350  3257  32100  35697  249888  62750  103286  1 73830  589754  212059  801813 

31  2355  3271  32100  34618  246988  69858  107459  167401  591706  256002  847708 

32  3003  3262  32100  33451  235555  70632  100094  163059  569348  184430  753770 

36  3025  3264  32100  28633  203392  67802  79920  138825  489939  84790  574729 

37  3032  3269  35300  31422  224856  71440  86377  152957  535630  204941  740571 

39  3044  3236  32100  23285  168276  69873  67234  113876  419259  126889  547148 

40  3054  3271  38300  34506  247579  87221  101453  167300  603553  307551  911104 

38  3038  3346  39136  36069  254080  92532  107306  171441  625359  354297  979656 

43  3067  3271  38300  34253  245704  99742  104659  164837  614942  344061  959003 

42  3064  3271  32100  30409  219235  74415  90447  145754  527851  207156  735007 

44  3073  3271  32100  28431  205083  79415  86477  135991  506966  271398  778364 

41  3059  3257  32100  27900  199042  76636  87533  135841  499052  173286  672338 
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197032 
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47 
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38300 

32216 

223585 

68251 

103476 

153152 

548464 

306557 

855021 

48 
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3351 

32100 

35687 
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168136 
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3117 

3361 

32100 
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82807 
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128731 

758898 

51 
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3341 

38300 

34107 

246276 

67423 

106340 

160110 
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3361 
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244312 

69805 

108173 
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54 
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K Sequence  Number 

ID  Induction  Date 

PD  Production  Date 

N Production  Load  Norm 

L Direct  Civilian  Man-Hours  Incurred 

L$  Direct  Labor  Costs  Incurred 

M$  Direct  Material  Costs  Incurred 

PE  Production  Expense  Applied 
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GA 
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APPENDIX  B 


Calculation  of  the  Index 
of  Standard  Hours 

The  index  of  standard  hours  for  a job  is  calculated  as  follows. 

For  any  given  job  its  induction  and  production  date  are  used  to 
establish  the  period  of  time  during  which  that  job  is  in  the  shop. 

The  standard  hours  produced  and  the  direct  man-hours  incurred  in  the 
production  of  those  standard  hours  in  the  953  shop  (aircraft  assembly) 
are  found  for  the  same  period  of  time.  The  ratio  of  these  standard 
hours  to  direct  man-hours  yield  a measure  of  the  efficiency  of  the  953 
shop  during  the  period  of  time  in  question.  The  direct  man-hours 

incurred  on  the  job  are  then  weighted  by  this  measure  of  efficiency  to 
produce  an  index  of  the  standard  hours  produced  in  the  accomplishment 
of  the  job. 

This  approach  is  subject  to  several  drawbacks  as  is  the  construc- 
tion of  any  index.  However,  since  the  vast  majority  of  the  work  in  the 
953  shop  is  on  F4-B  to  N conversions,  and  since  the  major  contribution 
of  direct  man-hours  to  any  B to  N job  is  from  the  953  shop;  this  pro- 
cedure seems  defensible. 
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APPENDIX  C 

Conditional  Estimation  and  Hypothesis  Testing 
With  Perfect  Multi  col  linearity^ 

Suppose  y = z6  + e (A.l) 

where  z = [X,  P],  X = [X] , X2],  X2  = [W,  (1-W)] 

(nxK+3)  * * 

6^  = [&]»  go! * $22’  Y-l 
(lxK*3) 

Let  P = [W,  (1-W)]  p 

where 

Thus  we  have  a problem  of  perfect  multi  coll inearity  since  p is  just  a 
linear  combination  of  W and  (1-W).  This  is  essentially  the  problem 
relating  the  dummy  variables  associated  with  WIPICS  and  the  material 
price  index. 

If  the  problem  is  reformulated  as  Theil  suggests  then: 

Y - Py  = Xb  + e (A.2) 

and  a conditional  estimator  may  be  derived  as 

^This  Appendix  follows  the  exposition  of  Theil  [10,  pp.  147-155]. 
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e = (xt  x)"1  xt  (y  - pY) 

= 8 - (X4  X)'1  xt  PY 


(A.  3) 


where  6 is  the  least  square  estimator  which  would  be  derived  by 
merely  dropping  P from  the  regression  equation.  That  is  8 is  the 
estimator  reported  in  (4.8). 


= X2  p = X 


[:] 


(A. 4) 


therefore 


6=6-  (Xt  X)"1  Xt  X f 0"I  y 


e21  ' y 


322  - 


(A. 5) 


Also  notice  that 


var  ( 8 ) = var  ( 6 ) = (xt  X)~^ 


(A. 6) 


and  it  can  be  estimated  by 


var  (e)  = s (X1  X) 


2 /ut  v\_l 


(A. 7) 


Thus  a conditional  estimator  for  6,  0 , may  be  derived  directly  from 

(A. 5)  given  values  for  £,  y,  and  P1.  Also  the  estimated  variance  of 
the  conditional  estimator  does  not  depend  on  the  values  of  y or  p-|. 

Finally  to  test  the  hypothesis  H0  : e21  - 622  = 0 a9ainst  the 
alternative  hypothesis  Ha  : ^21  " B22  > 0 we  need  only  calculate: 


where 


t = 


/'■ 


2 J 


(Xt  X)' 


a.1  = [0,  1,  -1]  and 

f ^ * * 

a e = 621  " y ~ 622  + pl  Y 


(A. 8) 


The  denominator  of  (A. 8)  is  just  the  estimated  standard  deviation  of 
c*  g.  This  is  the  basis  for  the  sensitivity  analysis  reported  in 
Section  5. 
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Air  Force  Institute  of  Technology 
Wright-Patterson  AFB,  OH  45433 

Logistics  policies  typically  assume  that  failure  times  are  generated 
from  a Poisson  process.  This  assumption  is  usually  caveated  as  being 
popular  or  a necessary  simplifation  for  mathematical  manipulation.  In 
addition,  major  assemblies  undergoing  extensive  repair  or  overhaul  are 
themselves  assumed  to  be  brand  new  upon  renewal.  The  influence  of  arbi- 
trary maintenance  policies  upon  the  observed  reliability  of  assemblies  has 
not  been  thoroughly  explored  at  present.  This  research  uses  a discrete 
event  probabilistic  simulation  model  to  examine  the  impact  of  five  realistic 
maintenance  policies  upon  nine  different  assemblies.  Although  the  study  is 
exploratory  and  the  conclusions  are  not  final,  indications  are  that  the 
hazard  function  for  an  assembly  does  indeed  pass  through  transient  condi- 
tions when  approaching  a steady  state.  Maintenance  policies  established 


as  a function  of  an  assembly's  hazard  function  must  then  change  over  time 
in  order  to  move  with  the  assembly's  changing  hazard  function. 

This  paper  is  not  included  in  the  proceedings  because  of  its  length. 
It  is  available  as  AD  #A030297  from  the  Defense  Documentation  Center, 
Cameron  Station,  Alexandria,  VA  22314. 
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SUMMARY 

A mathematical  structure  for  diagnostic  logic  modeling  was 
formulated,  which  allows  the  intrinsic  properties  of  a complex  Logic 
Model  to  be  studied  in  an  abstract  setting.  As  a result,  it  was  found 
that  a loop-free  Logic  Model  is  a partially  ordered  set  and  that  every 
permutation  of  the  elements  in  the  terminal  set  of  a finite  partially 
ordered  set  S partitions  S into  disjoint  subsets.  Based  on  these 
results,  it  was  deduced  that  the  minimum  number  of  test  points  required 
for  conclusive  detection  of  malfunctioning  components  for  a loop-free 
system  is  equal  to  the  number  of  elements  in  the  terminal  set;  this  set 
constitutes  the  optimal  choice  for  test  points.  Also,  it  was  estab- 
lished, for  each  permutation  of  the  elements  in  the  terminal  set,  a 
relative  failure  probability  measure.  Based  on  this  probability  meas- 
ure, an  optimal  diagnostic  strategy  was  defined  in  accordance  with 
Bellman's  Principle  of  Optimality.  Finally,  for  the  purpose  of 
illustration,  some  examples  are  given. 

INTRODUCTION 

Techniques  used  to  analyze  maintenance  characteristics  have  been 
frequently  subjective  and  difficult  to  determine  at  the  conceptual  and 
development  stages  of  an  equipment.  This  is  largely  due  to  the  lack 
of  analytical  methods  which  allow  the  functional  relationships  between 
the  hardware  components  or  parts  to  be  easily  understood.  Studies  (1), 
(2),  conducted  by  the  US  Army  Air  Mobility  R&D  Laboratory  in  reliability 
and  maintainability  technology,  address  this  problem  area. 

The  studies,  entitled  "Maintenance  Logic  Model  Analysis  Feasibility 
Investigation"  and  "LOGMOD  Diagnostic  Procedures  for  Mechanical  and 
Electrical  Equipment,"  address  the  feasibility  of  the  computer  generation 
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and  manipulation  of  a Logic  Model  Algorithm  for  use  as  a design  and 
maintenance  tool.  Included  are  diagnostic  procedures  for  assessing 
troubleshooting  and  maintenance  strategies  associated  with  hardware 
design. 

Automatic  data  processing  techniques  are  used  to  generate  an 
algorithm,  which  accepts  engineering  functional  design  data  and 
organizes  these  data  into  a structure  of  least  dependent  to  most  depend 
ent  hardware  elements.  The  structure,  called  tne  Logic  Model  (Diagram) 
is  precisely  a schematic  representation  of  the  intrinsic  functional 
relationships  between  the  elements  of  the  equipment.  The  Logic  Model 
was  developed  to  provide  improved  troubleshooting  analysis  and  evalua- 
tion of  hardware  design  concepts  for  maintenance  analysis  purposes. 

The  Logic  Model  concept  is  considered  an  engineering  innovation. 
Therefore,  a well  established  mathematical  basis  for  this  concept 
should  be  formulated.  This  then  allows  the  intrinsic  properties  of  a 
complex  Logic  Model  to  be  studied  in  an  abstract  setting.  Some  prelim- 
inary results  in  this  regard  have  been  published  (3).  The  purpose  here 
is  to  present  a mathematical  development  of  the  generic  concepts. 


PARTIALLY  ORDERED  SETS 


Well  established  mathematical  concepts  form  the  basis  upon  which 
the  techniques  discussed  here  are  based. 

Generally  speaking,  a partially  ordered  set  P is  a system 
consisting  of  a set  S and  a relation  'v  satisfying  the  following 
postulates: 

1.  a ^ b and  b Va  hold  if  and  only  if  a = b,  a,  b e S. 

2.  a ^ b and  b ^ c imply  a^c,  a,  b,  ceS. 

46 


m.  ■ s&s* 


If  S contains  only  a finite  number  of  elements,  then  P is 
called  a finite  partially  ordered  set.  In  this  paper,  we  are  con- 
cerned with  finite  partially  ordered  sets.  For  the  purpose  of  illus- 
tration, the  following  are  simple  examples  of  partially  ordered  sets: 

A.  Let  S = {1,  0,  -4,  10)  and  the  relation  *v  be  the  usual 
"greater  than  or  equal  to"  binary  relation.  The  system  is  a 
finite  partially  ordered  set,  for  obviously  the  set  S 
together  with  the  given  relation  satisfies  the  two  postulates 
above . 

B.  Let  S be  the  set  of  all  real  numbers  in  the  unit  interval 
with  the  relation  as  that  of  example  A.  Then  the  system  is 
a partially  ordered  set,  but  not  finite. 

C.  Let  S be  the  set  of  all  letters  in  the  word  "relation"  and 

the  relation  is  defined  by  the  arrangement:  relation,  that  is, 

e 'v  1 (read  e precedes  1);  this  is  a partially  ordered  set. 

However,  if  we  let  S be  the  set  of  all  lines  on  a plane  and  the  rela- 
tion be  "parallel;"  the  system  is  not  a partially  ordered  set  even 
though  it  satisfies  the  transitive  property  2 but  not  the  asymmetric 
requirement  1. 

BASIC  ELEMENTS  OF  THE  LOGIC  MODEL 

The  following  are  the  basic  elements  with  which  the  logic  model  is 
constructed.  For  a more  detailed  discussion  the  reader  is  referred  to 
the  published  reports  (1)  and  (2). 

Component  - A physical  component  of  an  equipment. 

Event  - A measurable  or  observable  quantity. 

Functional  entity  - A component  or  an  event. 

Dependence  - A functional  relationship  between  two  functional  entities. 
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Input  or  independent  event  - An  event  for  which  there  exists  no 
functional  entity  on  which  the  event  is  dependent. 

Terminal  event  - An  event  upon  which  no  functional  entity  depends. 
Dependence  chain  - A collection  of  functional  entities  in  which  every 
two  of  its  entites  are  dependent. 

Loop  - A closed  dependence  chain. 

SOME  INTRINSIC  PROPERTIES 

In  this  section  we  shall  state  and  prove  some  fundamental  character- 
istics of  the  Logic  Model.  The  mathematical  development  of  the  model 
is  an  elementary  exercise;  for  it  merely  translates  the  concepts  into 
an  algebraic  structure.  This  mathematical  structure  forms  a basis,  from 
which  some  interesting  results,  that  are  useful  for  maintenance  analysis, 
are  deduced. 

Lemma  1.  A loop-free  functional  logic  model  is  a partially  ordered  set. 

Proof:  Let  S be  the  set  of  all  functional  entities  of  the  logic 

model  and  the  functional  dependence,  denoted  by  <,  be  the  relation  on 
S.  Then,  for  xQ,  yQeS,  xQ  < yQ  and  yQ  < xQ,  the  logic  model  input  data 
requirements  ensure  that  there  exist  xl5  x2,  x3,  . . . , % and 
yi,  y2t  y3,  . . .,  yn  in  S such  that  the  following  dependency  chains 
can  be  realized: 

xQ  < x;  < x2  . . . < x^  < yD 

and 

yo  < yl  < y2  * • * < yn  < xo 

So,  concatenation  of  the  two  chains  gives 

xQ  < xj  < . . . < xm  < y0  < yj  < . . . < yn  < x0 
Now  if  there  exist  two  distinct  elements  in  the  set 

A r (xi,yj I i * 0,1,2, . . . ,m,  j— 0,1,2,.  . *,n}  , 
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then  the  chain  is  closed;  hence,  it  is  a loop  which  contradicts  the 
loop-free  assumption.  Therefore,  all  elements  of  A are  identical. 

It  remains  to  be  shown  the  transitive  property  holds.  Let  x0,  yQ,  and 
zQ  be  elements  in  S such  that  xc  < y0  and  yQ  < zQ.  Using  the 
similar  technique  as  before,  one  obtains  xQ  < zQ. 

Lemma  2.  For  every  finite  partially  ordered  set,  the  terminal  set 
is  not  empty. 

Proof:  Let  S be  a finite  partially  ordered  set  and  T be  the 

terminal  set  of  S.  To  prove  Lemma  2,  assume  the  contrary.  Let  ajeS. 
Then  a^T  implies  there  exists  an  element  a2eS  such  that  a2  < a\. 
Now  a2^T  implies  there  is  a3eS  such  that  a3  < a2.  Thus,  continu- 
ing the  process  inductively,  one  obtains  a dependency  chain, 
ak  < ak-i  K • • • < a2  < al » where  k is  a positive  integer  and  is 
less  than  or  equal  to  the  number  of  elements  in  S.  But  a^  t a^,  for 
all  i = 1,2,.  . . ,k-l,  since  S is  a partially  ordered  set.  There- 
fore, a^  is  a terminal  point,  by  definition.  Hence,  the  conclusion 
follows. 

Lenina  3.  Each  permutation  of  the  elements  in  the  terminal  set  T of 
a finite  partially  ordered  set  S partitions  S into 
disjoint  subsets  whose  union  is  precisely  S. 

Proof:  By  Lemma  2,  T contains  k elements,  for  some  positive 

integer  k.  Now,  let  the  ordered  k-tuples  (aj,  a2,  a3,.  . .,  a^)  be 
a permutation  of  the  elements  of  T.  Also,  we  define 


and 


49 


Hai  --  jxeS 


i-1 
U Hg 
4=0 


< x , 1 = 1,2,.  . .,k. 


Note  that  Hai  are  subsets  of  S,  and,  if  k = 1,  the  Lemma  3 conclu- 
sion is  obvious.  On  the  other  hand,  let  m,ne{l,2,3,.  . .,k}  such  that 
m j n and 

»am 

So,  let  yeHam  0 H3n.  Then  yeHam  and  yeH^.  Without  loss  of  general- 
ity, assume  m < n.  This  implies  that  there  exists  an  integer 
je{l,2, . . .,k-l}3n  = m+j.  By  definition. 


H, 


n-i 

xeS  - U H, 
i=0 


*i 


an  < x 


It  follows  that  ycHan  implies 

n-i  nH-j-1 

yi  U Ha^  = (j  Ha,  3 
i=0  i=o 

Therefore,  y^Ham,  which  contradicts  the  assumption  that  Han  and  Ham 
have  nonempty  intersection.  Hence,  these  subsets  are  all  disjoint.  It 
remains  to  be  shown  that  their  union  is  equal  to  S.  Suppose  xeS. 

Then  either  xeT  or  xeS-T.  If  xeT,  then 

k 

xe  U Ha . . 
i=  1 

If  xeS-T,  then  x^T.  x^T  implies  there  exists  a dependency  chain 
which  can  be  described  as 

x < X!  < x2  • . . < Xq_ j < aq  , 

for  some  integer  qe{l,2,.  . .,k}.  Therefore,  xeH^.  Also,  since  the 
union  of  subsets  is  again  a subset,  we  have 
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S - U Ha.  , 
i=l 

which  completes  the  proof  of  Lemma  3. 

Having  these  results  at  our  disposal,  it  is  a simple  exercise  to 
deduce  the  following  consequence  pertinent  to  diagnostics  as  applies 
to  analyzing  hardware  maintenance  and  design  concepts. 

Theorem  1.  The  minimum  number  of  test  points  required  for  conclu- 
sive detection  of  malfunctioning  components  for  a loop-free  functional 
system  is  equal  to  the  number  of  elements  in  the  terminal  set.  This 
set  constitutes  the  optimal  choice  for  test  points. 

Proof:  Lemma  3 implies  that  any  malfunctioning  entity  in  the  set 

S is  detectable  by  testing  the  terminal  points.  Also,  malfunctioning 
of  any  terminal  point  is  not  detectable  by  testing  the  other  terminal 
points  or  events. 

Before  we  continue,  let  us  examine  the  content  of  Theorem  1.  The 
apparent  drawback  of  Theorem  1 is  its  limitation  to  systems  not  con- 
taining a functional  dependency  loop.  For  system  containing  loop,  the 
conclusion  of  the  theorem  is,  in  general,  false  unless  the  loop  can  be 
degenerated  into  a functional  entity  which  is  not  a terminal  point. 

For  systems  containing  a functional  loop  whose  degeneration  gives  rise 
to  a terminal  point,  a remedy  for  this  problem  is  to  add  an  additional 
test  point  on  the  loop;  for  then  the  theorem  is  applicable  to  the 
modified  system. 

Although  Theorem  1 provides  some  quantitative  information  for 
conclusive  detection  of  system  malfunctioning,  no  information  can  be 
deduced  from  it  concerning  some  "optimal"  testing  sequence  on  the 
terminal  events.  In  what  follows,  an  attempt  is  made  to  analyze  a 


test  procedure  for  the  case  in  which  the  terminal  set  contains  two  or 
more  elements.  In  particular,  we  propose  to  determine  a diagnostic 
testing  sequence  (or  strategy)  for  conclusive  detection  of  system  mal- 
functioning such  that  the  expected  expenditure  of  resources  is 
minimum. 

For  the  analysis  of  this  problem,  we  observe  that  each  functional 
entity  of  a logic  model  relating  to  a hardware  component  or  measurable 
item  is  normally  considered  as  a potential  source  of  failure,  and 
therefore  a measure  of  failure  is  required  to  determine  the  relative 
failure  probability  associated  with  the  affected  functional  entities  in 
the  model.  With  this  in  mind,  we  state  and  prove  the  following 
statement : 

Theorem  2.  Each  permutation  of  the  elements  in  the  terminal  set 
induces  a relative  failure  probability  oo  the  corresponding  partition 
of  a finite  partially  ordered  set. 

Proof:  As  before,  let  T be  the  terminal  set  containing  k 

elements,  and  (aj,  a2,  a3, . . .,  a^)  be  a permutation  of  the  elements 
in  T.  Also,  let 

HA  s {Hai}  , 

where  i = 1,2,3,.  . .,k,  and  Hai  defined  as  before.  Now,  for  each 
xeHa^,  let  p^  be  the  failure  probability  of  the  functional  entity  x. 
Then  the  relative  failure  probability  Pai  associated  with  the  set 
Ha^  is  defined  to  be 


Now,  we  define  a function  on  by  the  formula 
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In  view  of  Lemma  3,  we  have 


ua(Ha)  = 1 . 

and  obviously, 

0 i bA(E)  i 1 , 

for  any  subset  E of  the  power  set  of  HA.  Hence,  the  assertion  holds, 
since  the  function  uA  so  constructed  is  a relative  failure  probability 
defined  on  the  sample  space. 

The  failure  probability  pa  constructed  in  the  proof  of  Theorem  2 
depends  on  the  permutation  A,  and  UA(Hai)  is  the  relative  probability 
of  observing  a failure  belonging  to  Hai  by  testing  the  test  point  a^ 
Therefore,  the  failure  probability  is  a function  of  the  testing  sequence 
and  the  component  failure  probabilities. 

Now  we  are  in  a position  to  define  an  optimal  strategy  for  conclusive 
detection  of  system  malfunctioning.  To  this  end,  we  cite  Bellman's 
Principle  of  Optimality  from  literature  (A)  as  follows: 

An  optimal  policy  has  the  property  that  whatever  the  initial 
state  and  initial  decision  are,  the  remaining  decisions  must 
constitute  an  optimal  policy  with  regard  to  the  state 
resulting  from  the  first  decision. 

Also,  we  define:  An  admissible  strategy  A is  that  strategy  such  that 

the  expected  expenditure  (cost  or  time)  incurred  due  to  applying  A to 
a given  functional  system  in  minimum.  Symbolically, 

k 

Min  £ Ua  P 
Ac A i-1 

where  A = (aj,  a2,  a3,  . . .,  a^) 


= terminal  point 

A = the  set  of  all  possible  strategies 

Uai  = the  expenditure  associated  with  the  test  point  a^ 

P3i  = the  relative  failure  probability  of  all  the  dependency  chains 
terminated  at  a^ 

An  optimal  strategy  is  an  admissible  strategy  which  obeys  the  Principle 
of  Optimality. 

EXAMPLE  MODELS 

A pictorial  representation  of  a logic  model  can  be  depicted  by 
using  arrows  and  points.  An  arrow  is  used  to  indicate  functional  or 
physical  dependency  between  two  functional  entities,  and  a point  indi- 
cates a particular  entity.  A working  logic  model  can  be  complex  con- 
taining several  hundred  functional  entities  and  dependencies,  however, 
only  a simple  model  is  needed  for  illustration  of  the  concepts  involved. 

Figure  1 shows  an  example  of  an  intermediate  loop  in  a model  which 
does  not  include  any  terminal  points.  This  is  a three  point  loop.  In 
general,  any  loop  can  involve  two  or  more  points.  In  Figure  2,  a 
terminal  loop  is  illustrated  and  involves  four  points.  If  the  loop 
were  to  degenerate  to  a point  the  point  would  be  a terminal  event  for 
the  model.  If  terminal  loops  exist  in  a design  it  will  often  mean 
that  troubleshooting  and  diagnosis  of  the  hardware  when  malfunctioning 
can  be  challenging  if  not  impossible  without  replacing  the  entire 
piece  of  equipment. 

A third  example  is  illustrated  by  Figure  3 to  illustrate  a sample 
calculation  for  a loop-free  system.  The  numerical  values  at  each 
point  are  used  to  represent  a measure  of  failure  probability  for  the 
functional  entity  represented  by  the  point.  The  three  terminal  points 
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which  exist  are  labeled  simply  as  (T) , (3),  and  (5),  respectively.  For 

this  case  then  there  are  a total  of  3!  strategies  possible  for  executing 

a test  procedure  to  detect  a malfunctional  entity.  These  are: 

TERMINAL  POINT 

STRATEGY  TEST  SEQUENCE 
2 -►  1 -*■  3 


A 

B 

C 

D 

E 

F 


1 -+  2 -*•  3 

3 -*■  1 -*■  2 

3 2 -*■  1 

2 -*•  3 -*■  1 

1 -»  3 -»■  2 


For  strategy  A the  partitioned  sets  of  elements  which  exist  are  indi- 
cated by  the  dashed  lines.  Figure  4.  This  partitioning  would  change 
for  each  strategy  and  therefore  the  corresponding  probability  for 
detection  of  a malfunction  at  each  test  point  will  change  with  each 
strategy. 

The  probabilities  of  detection  and  expected  expenditures  for  each 
strategy  (A  through  F)  are  presented  in  Table  1 and  Table  2.  The  proba- 
bility of  detection  at  each  test  point  is  listed  in  the  strategy  column 
and  the  entries  within  the  table  show  the  expected  expenditure  in  terms 
of  cost  and  time,  respectively,  in  the  two  tables.  The  numerical  values 
associated  with  expenditures  at  each  test  point  are  given  at  the  bottom 
of  each  table.  Note  that  in  Table  1 there  are  three  strategies  which 
yield  a minimum  value  of  1.08.  These  are  considered  admissible  strat- 
egies as  previously  discussed.  However,  only  strategy  F is  considered 
optimal.  Therefore,  on  a cost  basis,  this  is  the  best  choice  for 
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testing.  However,  on  a time  basis,  strategy  A is  optical  as  shown  in 
Table  2 This  is  an  interesting  situation.  A time-cost  tradeoff 
must  be  made  if  a single  best  strategy  is  to  be  chosen. 

Figure  5 plots  the  values  of  expected  time  versus  expected  cost 
for  each  strategy  and  indicates  the  overall  preferred  strategy  to 
be  B. 

This  type  of  analysis  is  normally  performed  by  a computer  for 
models  of  any  significant  size  where  the  number  of  strategies  grows 
factorially  with  the  number  of  test  points. 

CONCLUSIONS 

The  logic  modeling  concept  is  considered  to  be  an  engineering 
innovation.  In  this  paper,  the  mathematical  structure  for  a class  of 
logic  models  was  established,  and  some  of  its  generic  properties,  use- 
ful for  maintenance  analysis,  were  deduced.  Specifically,  we  make  the 
following  concluding  remarks: 

1.  A loop-free  functional  logic  model  is  a partially  ordered  set. 

2.  For  every  finite  partially  ordered  set,  the  terminal  set  is 
not  empty. 

3.  Each  permutation  of  the  elements  in  the  terminal  set  T of  a 
finite  partially  ordered  set  S partitions  S into  disjoint  subsets 
whose  union  is  precisely  S. 

4.  The  minimum  number  of  test  points  required  for  conclusive 
detection  of  malfunctioning  components  of  a loop-free  functional  sys- 
tem is  equal  to  the  number  of  elements  in  the  terminal  set.  This  set 
constitutes  the  optimal  choice  fc  r test  points. 

5.  The  conclusion  of  statement  4 is  valid  for  systems  containing 
loops  only  if  the  loop  can  be  degenerated  into  a functional  entity  which 
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is  not  a terminal  point.  If  the  degenerated  entity  is  a terminal 
point,  then  statement  4 is  applicable  if  the  system  is  modified  to 
allow  an  additional  test  point  in  the  loop. 

6.  Each  permutation  of  the  elements  in  the  terminal  set  induces 
a relative  failure  probability  lor  the  corresponding  partition  of  S. 

7.  Based  on  statement  6,  an  admissible  diagnostic  strategy  A is 
defined  to  be  a strategy  such  that  the  expected  expenditure  (cost  and/or 
time)  incurred  due  to  applying  A to  a given  functional  system  is  a 
minimum. 

8.  An  optimal  strategy  is  an  admissible  strategy  which  obeys  the 
Principle  of  Optimality. 

9.  An  optimal  strategy  for  a given  system  need  not  be  unique. 
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TABLE  2.-  EXAMPLE  OF  TIME  COMPUTATION 


Strategy 


Expected  time 
1st  T.P. 


Exp.  time  1st  Exp.  time  1st  and 

and  2nd  T.P.  2nd  and  3rd  T.P. 


A 2 + 1 + 3 .36  1.00 

.36  .32  .32 

B 1 + 2 + 3 .80  1.08 

.40  .28  .32 

C3+1+2  2.52  2.68 

.84  .08  .08 

D3+2+1  2.52  2.60 

.84  .08  .08 

E 2 + 3 + 1 .36  2.04 

.36  .56  .08 

Fl  + 3 + 2 .80  2.36 

.40  .52  .08 


1.96 


2.04 


2.76 


2.76 


2.20 


2.44 


For:  Tj  = 2,  T2  = 1,  T3  = 3 
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INTRODUCTION 


The  focus  of  this  paper  is  the  support  of  the  software  element  of  auto- 
matic test  equipment  systems,  or  ATE  systems.  We  define  support  to  be  the 
change  and  correction  of  discrepancies  which  is  a constant  requirement  for 
ATE  test  software.  Test  software  is  computer  programs  in  the  ATE  hardware 
which  cause  it  to  measure  critical  parameters  of  the  item  being  tested. 

Over  the  last  few  years,  an  increasing  number  of  Air  Force  organiza- 
tions have  taken  positive  steps  to  improve  the  acquisition  and  support  of 
the  software  used  with  ATE.  Conferences  have  been  held.  Studies  have  been 
conducted.  New  organizations  have  been  formed,  and  regulations  have  been 
published.  Without  question,  important  progress  has  been  made  in  solving 
some  problems  and  in  bringing  others  into  better  focus.  But  many  of  the 
resulting  recommendations  and  actions  have  dealt  in  generalities  or  abso- 
lutes. The  recommendations  of  major  studies  in  this  area  tend  to  imply 
that  specific  courses  of  action  are  applicable  in  all  situations.  The 
usual  recommendations  are  to  buy  more  documentation,  more  training, 
standardize  the  programming  language,  and  expand  the  organic  support 
capability. 

The  genesis  of  this  research  was  our  belief  that  "acquisition  managers" 
have  had  little  visibility  into  the  long  term  impacts  of  their  decisions  in 
the  ATE  software  area.  They  are  driven  very  strongly  by  near  term  costs 
because  of  the  way  the  Department  of  Defense  acquisition  "system"  works  and 
because  there  was  no  data  available  on  long  term  effects.  This  research 
doesn't  alleviate  the  first  constraint,  but  does  provide  a tool  which  will 
help  in  the  second  area. 


THE  APPROACH 

Our  premises  are  (1)  that  ATE  software  support  (the  cost  and  respon- 
siveness characteristics)  is  influenced  by  factors  subject  to  decisions 
made  during  the  acquisition  phase  (the  ATE  software  acquisition  management 
interfaces  are  shown  in  Figure  1),  and  (2)  that  the  support  of  ATE  software 
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(the  activities  of  making  necessary  corrections/modifications  to  it)  can 
be  modeled  as  a set  of  interrelated  processes  with  definable  attributes, 
e.g.  timeliness,  cost.  With  these  premises,  the  general  approach  was  first 
to  hypothesize  influencing  factors  which  were  investigated,  analyzed,  and 
tested.  Then,  the  Air  Force  support  process  was  modeled  to  include  these 
influencing  factors  which  come  from  early  acquisition  decisions. 

There  are  some  aspects  of  the  problem  which  seem  obvious.  Clearly, 
the  decision  to  provide  organic,  contractual,  or  mixed  organic  and  contract 
support  of  the  software  is  crucial.  Most  people  also  recognize  that  the 
decisions  on  test  program  documentation  will  affect  the  long  term  support 
capability.  The  problems  which  need  resolution  are  those  related  to  the 
questions  of  how  much,  and  under  what  circumstances.  In  the  case  of  ATE 
software  support,  the  questions  do  not  have  unequivocal  or  simple  answers. 
The  interacting  policies,  decisions,  and  processes  related  to  ATE  software 
maintenance  form  a complex  support  system.  It  is  this  complexity  which  is 
the  heart  of  the  problem. 

System  modeling  is  one  approach  to  dealing  with  complexity.  Where 
narrative  and  analytical  models  are  so  complicated  that  solution  of  the 
model  rivals  the  difficulty  of  less  formal  approaches,  simulation  tech- 
niques offer  an  alternative.  This  research  has  been  directed  toward  the 
development  of  a general  simulation  model  which  allows  managers  associated 
with  ATE  software  acquisition  and  support  to  investigate  the  probable  out- 
comes of  adopting  one  of  several  alternative  courses  of  action. 

INFLUENCING  FACTORS  DERIVED  FROM  ACQUISITION  DECISIONS 

There  were  three  factors  which  derive  from  acquisition  phase  decisions 
which  we  hypothesized  would  affect  ATE  software  support.  A major  portion 
of  the  research  effort  was  devoted  to  deriving  a relationship  which  would 
describe  the  effects  which  programming  language  level  (L),  the  amount  of 
documentation  (D),  and  the  amount  of  training  programmers  received  (T) 
have  on  the  time  required  for  programmers  to  complete  software  support 
tasks.  It  was  necessary  to  determine  the  form  of  the  effect  and  to  imple- 
ment a mathematical  description  of  it  in  the  simulation  model. 
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We  hypothesized  three  levels  for  each  of  these  influencing  factors,  as 
follows. 

Language  Levels 

H = Standard  Problem-Oriented  Language 
M = Nonstandard  Problem-Oriented  Language 
L = Tester=Oriented  Intermediate  Language 

Documentation  Levels 

H = Complete,  for  example  Test  Requirements  Document 
M = Intermediate,  listings  and  flow  charts 
L = Minimum,  only  listings 

Training  Levels 

H = four  to  six  weeks  training 
M = two  to  three  weeks  training 
L = one  week  or  less  training 

Data  was  collected  from  Air  Force  programmers  with  ATE  software  support 
experience  on  how  these  different  factor  levels  affected  their  "time  to 
accomplish  a service  action  on  a typical  job."  Using  this  data,  adjust- 
ments were  derived  using  regression  analysis  for  different  conditions--a 
situation  with  a specific  level  of  each  factor.  This  LDT  adjustment  was 
incorporated  into  the  support  process  model. 

THE  MODEL 

This  model  was  developed  based  upon  extensive  observation,  interviews, 
and  data  gathering  within  the  Air  Force  ATE  software  support  systems  at 
Ogden,  Warner  Robins,  and  Sacramento  Air  Logistic  Centers.  The  model  may 
be  viewed  as  a network  of  events  through  which  transactions  (discrepancy 
reports,  work  orders,  contract  calls,  corrected  tapes,  etc.)  flow.  The 
model  treats  each  type  of  transaction  as  one  form  of  the  service  demand. 
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Figure  2 shows  the  very  basic  schematic  form  of  the  model.  Service  de- 
mands enter  the  model  on  a random  basis  as  determined  by  the  Demand  Creation 
and  Classification  Segment.  After  a routine  (none-ECP)  demand  is  created, 
it  passes  through  the  Initial  Processing  and  Support  Source  Selection  segment. 

Demands  identified  as  ECP-related  can  be  moved  directly  to  contractor  support. 

The  support  source  selected  for  routine  demands  is  based  on  several  alterna- 
tive support  policies  which  may  be  specified  by  the  model  user.  From  this 
point  on,  the  processes  which  any  demand  encounters  are  quite  different 
depending  upon  the  organic/contract  decision.  When  contract  support  is 
selected,  costs  and  time  delays  are  incurred  for  preparation  of  a statement 
of  work,  funding  approvals,  procurement  processing,  contractor  services,  and 
technical  order  printing  and  distribution.  If  a demand  is  organically  ser- 
viced, delays  and  costs  are  incurred  for  preparation  of  a work  order,  com- 
puter support,  programmer  efforts,  technical  order  editing,  negative  prepar- 
ation, printing,  and  distribution.  Within  the  organic  segment,  there  are 
a maximum  of  twenty  programmers.  Each  programmer  is  modeled  separately. 

Within  this  segment,  task  size  is  determined  as  an  observation  of  random 
variable  and  the  language,  documentation,  and  training  effects  are  included. 

A time  dependent  on-the-job  training  effect  also  can  be  used. 

The  model  operates  on  a time  base  where  the  smallest  increment  is  one 
hour,  representing  one  working  hour  of  an  eight  hour  day,  five  day  week. 

The  user  specifies  the  number  of  years  the  simulation  runs  and  the  starting 
points  of  the  random  number  generators. 

A large  amount  of  data  can  be  obtained  from  a model  run.  Data  is  pro- 
vided in  four  classes:  (1)  model  parameters  used  in  the  simulation  run, 

(2)  quarterly  data,  (3)  annual  statistics,  and  (4)  end  of  simulation 
statistics.  To  illustrate  annual  statistics,  the  average  number  of  demands 
in  the  system,  the  cost  of  organic  programmer  services,  the  cost  of  organic 
idle  time,  among  others,  are  provided.  At  the  end  of  the  simulation,  all 
cost  and  time  data  are  summarized. 

Viewed  from  this  perspective,  the  operation  of  the  model  is  extremely 
straightforward.  It  is  within  the  modules  and  segments  that  the  essential 
characteristics  of  the  simulation  are  contained.  A complete  description  of 
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this  model  and  its  computerized  implementation  may  be  found  in  a recent 
AFIT  document  which  is  Reference  1. 

SO  WHAT 

The  result  of  this  research  is  a simulation  model  of  the  Air  Force  ATE 
software  support  process  which  incorporates  the  influences  of  factors  whose 
"levels"  are  determined  primarily  in  the  presupport  acquisition  phase. 

This  capability  will  permit  acquisition  managers  to  weigh  the  effects  of 
the  alternatives  to  the  decisions  they  must  make  concerning  the  support  sys- 
tem. Additionally,  since  the  Air  Force  ATE  software  support  process  is 
modeled,  the  effects  of  changes  within  that  process  can  be  evaluated. 

The  limitations  associated  with  this  capability  must  also  be  understood. 
The  Air  Force  ATE  software  support  system  was  modelled,  not  any  support 
system.  The  data  used  to  specify  the  parameters  and  the  random  variables  in 
the  system  model  were  thoroughly  researched  but  were  incomplete  in  some 
instances  and  of  poor  fidelity  in  some  other  cases.  The  sources  of  the  data 
and  specific  cautions  are  included  in  Reference  1.  Model  validation  has 
been  limited  to  a "face  validity"  effort  reviewing  the  functional  form 
and  sample  outputs  with  experienced  personnel  within  the  Air  Force  ATE  soft- 
ware support  system. 

We  think  this  specific  methodology  might  be  used  in  the  following  appli- 
cations related  to  initial  acquisition. 

(1)  Investigating  the  desirability  of  developing  a standard  ATE 
programming  language 

(2)  Selecting  a language  level  for  a new  test  system  development 

(3)  Selecting  a level  of  ATE  software  documentation  for  a particu- 
lar program 

(4)  Selecting  a training  level  for  organic  ATE  programmers 

(5)  Selecting  an  ATE  software  support  concept 

Even  after  the  initial  acquisition  decisions  have  been  made,  there  are 
times  when  this  methodology  can  be  of  assistance  to  Air  Force  managers. 

(1)  Investigating  budgeting  requirements  for  organic  software  support 

(2)  Manpower  planning  for  organic  ATE  software  support 


(3)  Investigating  mixes  of  organic  and  contract  support  when 
manpower  resources  are  limited. 

From  a broader  perspective,  we  feel  that  the  type  of  research  re- 
ported here  may  have  much  expanded  application  than  simply  the  Air  Force 
ATE  software  support  system.  Modelling  efforts  aimed  at  developing 
understanding  of  the  repair  and  support  processes,  especially  the  inter- 
actions between  presupport  decisions  and  the  resulting  processes,  can 
help  provide  acquisition  managers  insight  into  probable  future  implica- 
tions of  decisions  among  alternatives  being  weighed  today.  These  are  the 
kinds  of  things  which  must  be  done  if  we,  the  Department  of  Defense,  are 
ever  going  to  do  more  than  pay  lip  service  to  "life  cycle  costs." 

EXAMPLE  APPLICATIONS 
Analysis  of  an  Acquisition  Problem 

To  demonstrate  the  application  of  the  methodology  to  consideration  of 
alternatives  which  might  be  of  interest  during  the  acquisition  of  ATE 
software,  a sample  problem  was  constructed  and  analyzed.  The  purpose  of 
the  problem  was  to  demonstrate  how  the  model  could  be  used  to  provide 
information  bearing  on  a decision  to  develop  an  organic  capability  and  in 
selecting  a documentation  level.  In  the  problem  the  following  assumptions 
were  made: 

(1)  The  test  programs  are  used  with  a family  of  intermediate 
level  ATE  used  to  test  avionics  Line  Replaceable  Units 
(LRU’s). 

(2)  There  are  350  test  programs  to  be  supported. 

(3)  The  programming  language  is  a problem-oriented  language, 
but  peculiar  to  this  family  of  ATE. 

(4)  Funds  are  available  for  only  a week,  introductory  training 
course. 

The  problem  which  is  posed  is  two-fold.  First,  should  organic  or 
contractor  support  be  adopted?  And  second,  if  organic  support  is  reccom- 
mended,  what  level  of  documentation  should  be  acquired?  The  contractor 
has  provided  the  following  information: 
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--Delivery  of  the  required  compiler  and  the  associated  compiler 
documentation  will  cost  $440,000. 

--Delivery  of  program  listings  alone  will  cost  $500  per  program. 

--Delivery  of  program  listings  and  flow  diagrams  will  cost 
$2,000  per  program. 

--Delivery  of  Test  Requirements  Documents  (TRD's)  and  Program 
listings  will  cost  $10,0CC  per  program. 

To  investigate  this  problem,  a four-step  procedure  was  used.  First, 
the  model  parameters  were  set  to  establish  the  desired  characteristics. 
Second,  the  model  was  run  for  four  different  situations--once  for  total 
contractor  support,  one  time  each  for  organic  support  of  routine  demands 
with  low,  medium,  and  high  levels  of  documentation.  Next,  the  model  re- 
sults were  analyzed.  As  a final  step,  the  support  impacts  were  compared 
with  the  initial  acquisition  costs.  As  a note,  the  simulation  was  repli- 
cated three  times  for  each  of  the  four  situations.  This  was  done  to 
determine  the  effect  which  the  random  processes  in  the  model  had  on  the 
data. 

Table  1 shows  the  results  in  terms  of  the  total  support  costs  for 
eight  years  of  operation  for  each  policy.  Statistical  hypothesis  tests 
allowed  rejection  of  the  null  hypotheses  that  these  costs  were  identical 
for  any  pair. 

Adding  the  original  documentation  costs  plus  the  compiler  costs,  the 
total  costs  are  shown  in  Table  2.  This  calculation  of  total  acquisition 
and  support  cost  has  been  computed  without  considering  the  time  value  of 
money  concept.  Discounting  was  not  used  to  keep  the  example  simple. 

Wtih  these  calculations,  the  policies  can  now  be  ranked:  Policy  IV  > 

Policy  III  > Policy  I > Policy  II. 

Other  information  can  be  obtained  from  the  simulation  which  should 
be  considered  along  with  the  cost  data.  These  data  are  those  related  to 
the  responsiveness  of  the  support.  One  potentially  key  parameter  is  the 
average  time  taken  to  satisfy  a routine  demand.  This  value  is  identified 
in  Table  3 for  each  policy. 
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If  a graph  such  as  Figure  3 is  used,  one  can  visualize  both  cost  and 
responsiveness  aspects  of  each  of  the  four  policies.  When  examined  only 
in  terms  of  total  costs,  there  is  not  a large  difference  in  Policies  I 
and  II.  Likewise,  Policies  II  and  III  would  lead  to  about  the  same 
average  time  to  complete  a routine  demand.  But,  when  cost  and  responsive- 
ness are  viewed  together,  Policy  II  seems  to  be  the  one  which  would  be 
preferred.  This,  of  course,  is  based  on  the  assumption  that  the  decision- 
maker places  some  positive  value  on  a decrease  in  average  service  time  per 
demand. 

Analysis  of  Support  Manpower  Requirements 

The  second  example  was  chosen  to  demonstrate  one  way  in  which  the  ATE 
Software  Support  Simulation  Model  can  be  used  to  investigate  organic  man- 
power assignment  trade-offs.  In  this  example  all  of  the  model  parameters 
will  be  held  constant  except  the  number  of  programmers.  By  using  the 
model,  alternative  manpower  policies  will  be  examined  over  a two-year 
period. 

The  first  step  taken  in  examining  this  problem  was  to  run  the  simula- 
tion for  two  years  with  the  number  of  programmers  held  constant  at  20. 

This  was  done  to  obtain  the  quarterly  printout  of  the  average  number  busy 
in  each  quarter.  Us  ng  this  information,  several  alternative  policies 
were  hypothesized  and  examined. 

A total  of  ten  combinations  of  manpower  assignments  were  simulated. 
Selected  outputs  of  the  simulation  are  shown  in  Table  4.  The  results  here 
are  for  a single  experiment  for  each  policy.  Were  this  an  actual  problem, 
some  replication  for  each  policy  would  be  advised.  From  the  data  in 
Table  4,  a graph  was  constructed  similar  to  the  one  used  with  the  first 
example.  Here,  average  programmer  costs  per  demand  were  used  as  a measure 
of  the  monetary  impact  of  a policy.  Figure  4 shows  the  ten  combinations 
on  a single  graph.  No  attempt  will  be  made  to  identify  a best  policy. 

It  can  be  seen  that  significant  trade-offs  do  exist.  Lower  average 
service  times  can  be  obtained  for  the  price  of  an  increase  in  total  man- 
power comrvi  tment  and  average  cost  per  demand.  In  general  terms,  one  can 
c. .oose  any  one  of  several  alternatives  depending  on  the  objectives  which 
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one  desired  to  achieve  or  depending  upon  existing  constraints.  Without 
the  use  of  some  tool,  it  is  not  clear  how  such  trade-offs  could  otherwise 
be  considered. 

The  two  examples  presented  provide  a means  of  investigating  two 
hypothetical  ATE  software  acquisition  and  support  problems.  The  results 
of  the  simulations  provide  the  reader  a means  to  judge  whether  the  model 
does  provide  insights  which  were  not  otherwise  obvious. 
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Table  1 

Statistics  Related  to  Total  Costs  (Support) 
From  Simulation  Example  #1.  8 years 


! Policy 

Mean 

Standard  Deviation 

I 

Organic  Support  (DOC  = l) 

$6,244,736 

51 ,617 

II 

Organic  Support  (DOC  =2) 

5,540,136 

71,085 

in 

Organic  Support  (DOC  =3) 

4,990,622 

159,700 

IV 

Contractor  Support 

9,830,028 

87,488 

1 

(DOC  means  Documentation  Level) 

Table  2 

Acquisition  and  Support  Costs 
Simulation  Example  #1 


Policy: 

Documentation 

+ Compiler  + Support 

= Total  Cost 

I 

• 175,000 

550,000 

6,245,000 

6,970,000 

II 

700,000 

550,000 

5,540,000 

6,790,000 

ni 

3,500,000 

550,000 

4,991,000 

9,041,000 

IV 

-0- 

9,830,000 

9,830,000 

. dtmiii  twit  tir- 


Table  3 

Responsiveness  Data  Simulation  Example  #1 


toUcy 

Average  Time  Per  Demand  (Routine  only) 

I 

858  hours  or  5»0  months 

n 

627  hours  or  3.6  months 

m 

559  hours  or  3.2  months 

IV 

789  hours  or  4«5  months 

AVERAGE 

5* 

SUPPORT 

TIME  FOR 

ROUTINE 

4* 

MJS.WM'ii 


(MOUTHS) 


6 7 8 9 10 

TOTAL  8 YEAR  ACQUISITION  & SUPPORT 
COSTS  (MILLIONS) 


FIGURE  3 

Policy  Comparison  Example  #1 
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Ifeble  4 

Data  for  Simulation  Example  #2 


■tuber  of 
Programmers 

Avg.  Time 
Rout  ine 
Demands 

Avg.  Cost 
per 
Demand 

Avg.  Number 
in 

System 

Ir  1 

Ir  2 

(Mos) 

12 

9 

4*64 

2162 

78.0 

13 

8 

4.19 

2167 

72.0 

13 

9 

4.06 

2172 

70.0 

13 

10 

3.85 

2202 

67.0 

14 

8 

3.42 

2168 

61.5 

14 

9 

3.25 

2196 

59.5 

14 

10 

3.12 

2300 

57.5 

15 

9 

2.99 

2299 

56.0 

15 

10 

2.95 

2395 

55.0 

16 

10 

2.86 

2472 

54.0 
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FIGURE  4 

Policy  Comparison  Example  #2 
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ASSESSING  TEST  REPEATABILITY 
Girard  Levy  and  Horace  Ray 
Battelle  Columbu9  Laboratories,  Columbus,  Ohio 

One  of  the  general  problems  in  optimizing  maintenance  decisions 
is  the  determination  of  test  repeatability,  validity,  and  sufficiency. 
Validity  and  sufficiency  are  properly  determined  analytically  and  then 
verified  empirically.  One  approach  to  determining  validity  and  sufficiency 
involves  developing  an  analytical  model  relating  mission  performance  to 
values  of  various  system  test  parameters.  Much  useful  information  for 
maintenance  optimization  can  come  from  such  an  approach  (Reference  TASC 
reports) . 

Determination  of  test  repeatability,  on  the  other  hand,  is  not 
typically  accomplished  with  the  same  care  (exceptions  are  References  1 
and  2).  Usually,  the  specification  of  test  equipment  is  based  on  (a) 
known  (or  assumed)  measurements  precision,  and  (b)  some  feel  for  the 
relationship  of  variation  in  test  parameters  to  variation  in  mission 
performance.  Measurement  variations  must  be  significantly  less  than 
variations  due  to  other  sources,  and  less  than  variations  which  would 
affect  mission  performance. 

In  order  to  investigate  test  repeatability  and  isolate  some  of 
the  factors  which  might  lead  to  measurement  variation,  a small  data 
collection  effort  was  initiated  with  respect  to  a selected  test  point  in 
the  AN/ AS N- 90  depot  repair  process.  The  particular  test  which  was 
selected  for  this  purpose  was  the  final  acceptance  test.  This  test  was 
selected  because,  out  of  all  depot  repair  and  test  operations,  the  final 
acceptance  test  represents  the  single  largest  cost  item. 

The  data  collection  design  for  collecting  test  repeatability 
data  is  described  below,  followed  by  a presentation  of  selected  results 
from  implementing  the  design. 

Data  Collection  Design 

Development  of  the  data  collection  design  was  Influenced  by 
several  considerations.  First,  it  was  judged  important  that  any  design 
developed  provide  data  to  document  the  extent  to  which  there  is,  in  fact. 
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a test  repeatability  problem.  Toward  this  end,  any  data  collection 
design  developed  had  to  yield  data  that  would  provide  an  estimate  of  the 
total  unreliability  of  test  results,-  i.e.,  an  estimate  of  the  total 
variation  in  repeated  measurements  for  a given  component  (item)  due  to 
the  combined  influence  of  all  factors  in  the  system  which  might  produce 
such  variation. 

Second,  it  was  judged  important  that  any  design  developed 
provide  information  on  some  of  the  potential  sources  of  unreliability  of 
test  results.  These  potential  sources  included:  (1)  systematic  differ- 

ences between  test  stations  (due  to,  e.g.,  calibration  differences), 

(2)  systematic  differences  between  operators  at  test  stations,  and  (3)  for 
a given  operator  and  test  station,  measurement  variation  over  time  for  a 
given  component.  Systematic  differences  between  test  stations  and/or 
operators,  if  found,  would  suggest  alternative  corrective  actions  to 
possibly  reduce  such  differences,  such  as  improved  operator  training,  or 
improved  station  calibration  procedures.  On  the  other  hand,  variation  in 
test  results  for  a given  station  and  operator  could  be  attributable  to 
several  sources,  including  time  to  time  variations  within  the  component 
being  tested,  and/or  within  the  operator,  and/or  within  the  test  equipment 
itself.  Additional  studies  would  be  required  to  separate  out  these  sources 
of  variation. 

A third  major  consideration  in  developing  the  data  collection 
design  concerned  the  number  of  "extra"  tests  that  would  be  required,  i.e., 
the  number  of  tests  over  and  above  the  single  test  that  would  normally 
be  made  on  each  component.  In  turn,  the  total  number  of  extra  tests  is  a 
function  of  the  number  of  components  for  which  repeated  measurement  data 
are  to  be  obtained,  and  the  number  of  repeated  measurements  on  each 
component.  Clearly,  any  design  developed  had  to  hold  the  number  of  extra 
tests  within  reasonable  bounds,  in  view  of  the  time/costs  associated  with 
such  tests. 

A fourth  requirement  concerned  the  number  of  operators  and  test 
stations  that  would  be  involved  in  the  data  collection,  in  order  to  assess 
any  differences  between  operators  and  test  stations.  It  was  decided  that 
at  least  three  operators  and  three  test  stations  should  be  involved. 


Several  alternative  experimental  designs  were  generated  and 
evaluated  in  view  of  the  above  requirements  and  constraints.  One  design 
considered  initially  was  a complete  factorial  design,  with  operators,  test 
stations,  and  components  as  the  three  factors,  with  each  component  being 
tested  at  all  operator-station  combinations.  This  design  was  rejected, 
however,  due  to  the  prohibitively  large  number  of  extra  tests  that  would 
be  required,  given  a sufficiently  large  sample  of  components.  For  example, 
for  nine  components,  three  test  stations,  and  three  operators,  81  total 
tests  would  be  required  (even  without  measurement  replication  at  given 
station-operator  combinations)  with  72  of  these  tests  being  extra  tests. 

A second  design  considered  was  a Latin  Square  design,  with,  for 
example,  three  operators,  three  test  stations,  and  three  components,  with 
each  component  tested  at  three  different  station-operator  combinations. 

This  type  of  design  was  considered  undesirable,  however,  due  to  the  possible 
confounding  of  the  main  effects  of  stations,  operators,  and  components  with 
interactions  involving  these  factors.  Since  it  was  not  judged  tenable  to 
assume  that  these  interactions  were  zero,  this  design  was  rejected. 

A third  design  considered  was  a "randomized  groups"  design,  with 
n different  components  assigned  at  random  to  each  station-operator 
combination,  and  with  each  component  being  tested  twice  at  a given  station- 
operator  combination.  For  example,  with  three  operators  and  three  test 
stations,  three  different  components  might  be  assigned  at  random  to  each 
of  the  nine  station-operator  combinations.  Although  this  design  is 
statistically  sound,  it  was  judged  to  be  less  than  optimal  since  it  does 
not  provide  for  a direct  control  of  differences  between  components  in 
assessing  differences  between  stations  and  operators. 

The  data  collection  design  finally  selected  (see  Figure  1)  does 
provide  for  control  of  differences  between  components  in  assessing 
differences  between  stations  and  operators,  in  that  the  same  components 
are  tested  at  each  test  station,  and  the  same  components  tested  by  each 
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FIGURE  1.  DATA  COLLECTION  DESIGN 
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T 


operator.  The  design  calls  for  the  testing  of  a sample  of  nine  components 


(IMU's)  (C^  through  C^) , involving  three  test  stations,  and  one  operator 


selected  within  each  shift,  giving  three  operators. 

The  sample  of  nine  components  are  split  at  random  into  three 


groups  of  three  components  each  (Group  1 — C^,  C^,  C^;  Group  2 — C^,  C,., 


C ; Group  3 — C.,,  Cot  Cn).  Each  group  of  components  are  then  tested  at 
o / o y 


three  different  station-operator  combinations,  in  a way  such  that  all 
nine  components  are  tested  at  each  station,  and  such  that  all  nine  compo- 
nents are  tested  by  each  operator.  In  addition,  each  of  the  nine  components 
are  tested  twice  [indicated  by  the  (2)]  at  a given  station-operator 
combination,  to  assess  time  associated  differences  in  test  results. 

To  implement  the  design,  a total  of  36  final  acceptance  (ATP) 
tests  are  required  (each  of  nine  components  tested  four  times),  or  27 
"extra"  tests  over  and  above  the  single  tests  for  each  of  the  nine  compo- 
nents normally  required. 

The  test  schedule  for  implementing  the  design  if  shown  in 
Figure  2.  The  schedule  shows,  for  each  component  (IMU),  the  station- 
operator  combination  at  which  the  first,  second,  third,  and  fourth  tests 
occur.  The  schedule  is  designed  such  that  the  order  of  testing  is  balanced 
with  respect  to  test  stations  and  operators.  Also,  the  two  tests  for  a 
component  at  the  same  operator-station  combination  are  always  the  third 
and  fourth  tests  for  that  component. 

In  implementing  the  design,  for  the  first  three  tests  for  a given 
component,  only  a portion  of  the  ATP  was  to  be  completed  (approximately  the 
first  four  hours  of  testing).  Specifically,  the  first  three  tests  were 
to  begin  with  the  Gyro  Slew  Checks,  and  continue  through  the  calculation 
of  Gypto  Torquer  Scale  Factors  (see  Exhibit  A,  Data  Collection  Form).  The 


This  type  of  design  has  been  labelled  as  a Type  II  "mixed  design,"  See 
Lindquist,  E.  F.,  "Design  and  Analysis  of  Experiments  in  Psychology  and 
Education"  (Ref  4,  pp  273-281).  Also  see  Kirk,  R.  E.,  "Experimental 
Design:  Procedures  for  the  Behavioral  Sciences"  (Ref  2,  Chapter  5), 
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fourth  test  for  each  component  was  to  consist  of  the  full  ATP,  as 
ordinarily  conducted. 

The  above  "partial  testing"  plan  was  designed  to  reduce  time/ 
costs  associated  with  the  retesting  of  components.  However,  it  was 
judged  that  the  partial  tests  would  provide  sufficient  data  for  analysis 
purposes. 

The  analysis  of  variance  source  table  for  the  data  collection 
design  is  shown  as  Table  1,  which  gives  a breakdown  of  sources  of 
variation  considering  only  the  first  three  measurements  on  each  component. 
The  total  variation  is  broken  down  into  two  parts:  between  components, 

and  within  components.  The  between  components  variation,  in  turn,  breaks 
down  into:  (a)  variation  between  groups,  and  (b)  variation  between 

components  within  group's.  For  this  particular  design,  the  between  groups 
variation  reduces  to  the  between  component  portion  of  the  station  by 
operator  interaction,  i.e.,  that  part  of  the  operator  by  station  inter- 
action based  on  between  component  comparisons  [AB(b)]. 


* 


See  Lindquist,  (Ref  4). 


TABLE  1.  ANALYSIS  OF  VARIANCE  TABLE  FOR  DATA  COLLECTION  DESIGN 


Source  of  Variation 

df 

E(MS) 

Between  Components 

an- 1 

s 

8 

- G[AB(b)] 

a-1 

= 

2 

or2 

2 2 
+ ao'c  + na<r  q 

- C within  groups 

a(n-l) 

* 

6 

<r2 

+ acre 

Within  Components 

an (a-1) 

= 

18 

- A 

a-1 

= 

2 

<r  2 

+ narj 

- B 

a-1 

= 

2 

O'2 

2 

+ natr  g 

- AB (w) 

(a-1) (a-2) 

= 

2 

O'2 

+ n<rAB 

- Res idual 

a(a-l) (n-1) 

= 

12 

O'2 

Total 

2 . 
a n-1 

= 

26 

A * Test  stations 
B * Operators 
C = Components 
G * Groups 

a * Number  of  operators  = number  of  test  stations 
n * Number  of  components  in  each  group 
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The  within  components  variation,  which  Is  taken  as  a measure 
of  the  total  unreliability  of  test  results,  breaks  down  into;  (a) 
differences  between  test  stations  (A  -main  effect);  (b)  differences  between 
operators  (5  main  effect);  (c)  that  part  of  the  station  by  operator 
interaction  based  on  within  components  comparisons  I/lS(w)],  and  (d) 
residual  variation.  Part  of  this  residual  variation  is  variation  in 
repeated  measurements  of  a component  at  a given  station-operator  com- 
bination, which  can  be  calculated  based  on  the  third  and  fourth  measure- 
ments of  a component. 

The  last  column  of  Table  1 shows  the  variance  components  in 
the  expected  mean  squares.  As  illustrated  subsequently,  these  expressions 
can  be  used  to  calculate  variance  estimates  for  each  source  of  variation. 

In  summary,  repeated  measurement  data  was  to  be  collected  and 
analyzed  to  provide  results  that  include : 

(1)  Estimate  of  the  total  variation  in  repeated  measure- 
ments for  a given  component  due  to  the  combin'd 
influence  of  operator  factors,  test  equipment 
factors,  time  associated  variation  within  a 
component  (item)  being  tested,  and  other  unknown 
sources  of  variation 

(2)  Estimate  of  changes  in  "pass-fail"  decisions  as  a 
result  of  the  variability  in  (1)  above 

(3)  Estimate  of  measurement  variation  over  time  (for  a 
given  operator,  station,  and  component) 

(4)  Estimate  of  systematic  differences  between  test 
stations,  indicative  of  station  bias 

(5)  Estimate  of  systematic  differences  between  operators, 
indicative  of  operator  bias 

(6)  Estimate  of  differences  between  components  being 
tested 

(7)  Estimate  of  relative  variability  of  test  readings 
between  test  stations  and  between  operators. 
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The  above  estimates  were  to  be  obtained  for  each  of  the  test 
parameters  involved  in  the  ATP  for  which  data  are  being  collected.  These 
results  were  then  be  used  in  the  simulation  model  to  investigate  potential 
cost  savings  from  changes. 

Test  Results 

Test  data  were  collected  for  seven  different  measures  involved 
in  the  ATP.  Results  from  one  of  these  measures  (z  Slew  Rate)  are  pre- 
sented here.  Analogous  results  for  the  other  six  measures  are  presented 
in  Appendix 

Table  2 shows  the  analysis  of  variance  results  (associated 
means  and  standard  deviation  for  each  test  station  and  operator  are 
given  in  Table  3 ) . As  indicated  in  Table  2 , the  only  statistically 
significant  effect  was  the  main  effect  of  stations.*  The  operator  main 
effect,  and  the  operator  by  station  interaction,  both  were  not  statistically 
significant . 

However,  of  more  interest  for  this  problem  than  statistical 
significance  is  the  variance  estimates  for  each  source  of  variation. 

These  estimates  are  shown  in  Table  4. 

For  example,  the  variance  estimate  for  test  stations  is  13.490, 
This  estimate  was  calculated  using  the  expressions  for  the  expected  mean 
squares  given  earlier  in  Table  1,  as  follows: 

2 2 

(1)  MSa  = <r  + naer Substituting  in  this  expression  the 

2 

• calculated  values  for  MS ^ , O"  (residual),  and  values 

for  the  coefficients  n and  a.  gives: 

(2)  145.45  - 24.04  + (3)(3)<rj 

* The  error  term  in  the  F ratio  to  test  the  group  effect  [AB(b)]  is  the 
mean  square  of  components  within  groups.  The  error  term  in  the  F 
ratio  for  all  the  "within"  effects  is  the  residual  (within)  mean 
square . 
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TABLE 


, 


ANALYSIS  OF  VARIANCE  RESULTS 
FOR  Z AXIS  SLEW  RATE 


Source 

Between  Components 

• G [AB(b)  ] 

• C within  groups 

Within  Components 

• A 

• B 

• AB  (w) 

• Residual 

Total 


df 

SS 

MS 

F 

8 

13,563.29 

1,695.41 

2 

1,592.81 

796.40 

0.399 

6 

11,970.48 

1,995.08 

18 

724.88 

40.27 

2 

290.91 

145.45 

6.051 

2 

121.60 

60.80 

2.529 

2 

23.91 

11.95 

0.497 

12 

288.47 

24.04 

26 

14,288.17 

* 


i 

i 

1 


A “ Test  Stations 
B * Operators 
C * Components  (IMU's) 

G * Groups 

* 

F Ratio  significant  at  the  0.05  level. 
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TABLE  3 . 


MEANS  AND  STANDARD  DEVIATIONS  (Z  SLEW 
RATE)  FOR  EACH  OPERATOR  AND  TEST  STATION 


Standard 

Mean  Deviat ion 


Station  1 5392.7 

2 5400.7 

3 5396.9 


23.20 

24.51 

24.71 


Operator  1 


5398.8 


21.83 


TABLE  4 


VARIANCE  ESTIMATES  FOR  Z 
AXIS  SLEW  RATE 


Variance 

Percent  of 

Source 

Estimate 

Total  Variance 

Between  Components 

657.014 

94 

• G [AB (b) ] 

0.000 

• C within  G 

657.014 

Within  Components 

41.614 

6 

• A 

13.490 

2.0 

• B 

4.084 

0.6 

• A£(w) 

0.000 

0.0 

• Residual 

24.040 

3.4 

Total 

698.628 

100 

A * Test  Stations 
B - Operators 
C * Components  (IMU's) 
G = Groups 


n-T 


r 

J 


(3)  <r  ? - . HLi I . 13.490. 


The  other  variance  estimates  are  calculated  in  a similar  way.*  The  sum 
of  each  of  the  six  separate  variance  estimates  provides  an  estimate  of 
the  total  variance  (698.628), 

Inspection  of  these  estimates  shows  that  differences  between 


components  account  by  far  for  the  largest  portion  of  the  variance  - 


approximately  94  percent  of  the  total  variance  ( = 0.9404), 


693.628 

This  leaves  about  6 percent  of  the  variance  attributable  to  variation  in 

41 . 614 

repeated  measures  onto  (£QQ \oa  " 0.596).  That  is,  approximately  6 per- 


698.628 

cent  of  the  total  variance  is  due  to  repeated  measurements  of  a given 
component,  when  these  measurements  are  made  at  different  station/operator 
combination. 

If  the  total  variation  in  repeated  measurements  (41.614),  most 
of  this  variation  can  be  attributable  to  factors  other  than  systematic 


differences  between  test  stations  and  operators,  since  the  residual 


24.04 

variance  (24.04)  accounts  for  about  58  percent  (7,  * ) of  the  variation 


41.614 

in  repeated  measurements.  Differences  between  test  stations  account  for 


about  32  percent  of  the  variation  in  repeated  measurements,  and  differences 
between  operators  about  10  percent . 

Finally,  it  is  of  interest  to  estimate  the  variance  in 
repeated  measurements  for  a given  station/operator  combination,  i.e., 
the  variation  between  the  third  and  fourth  measurements  for  a given 
component,  as  pointed  out  earlier.  This  estimate  is  calculated  to  be 
15.632,  and  represents  the  residual  mean  square  in  a variance  breakdown 
with  operator  and  test  stations  as  two  factors,  with  two  measurements 
on  each  of  9 components  assigned  at  random — to  the  nine  station  operator 
combinations.  The  estimate  of  15.632  can  be  compared  with  the  residual 
variance  of  24.040  as  given  in  Table  2,  Indicating  that  over  one-half 


* Negative  variance  estimates  were  obtained  for  groups  (G)  and  AB(W ) . 
Thus,  the  value  of  0.0  was  used  for  each  of  these  two  estimates. 


- 
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of  the  latter  residual  variation  can  be  attributable  to  variation  in 
repeated  measurements  at  a given  station/operator  combination. 

As  mentioned  earlier,  the  various  estimates  such  as  given  above 
are  to  serve  as  imputs  to  the  simulation  model,  in  order  to  investigate 
potential  cost  savings  from  changes. 
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V. 


EXHIBIT  A . DATA  COLLECTION  FORM 


Start  ETI  reading 


hr 


9-27  Gyro  slew  check 


Z slew  rate  (OZ)  Actual  reading  

deg/hr 

X slew  rate  (OX)  Actual  reading  

deg/hr 

Y slew  rate  (OY)  Actual  reading  

deg/hr 

9-31  Azimuth  grid  mode  slew 

Eire  sinX  Actual  reading  deg/hr /V 


9-35  Multi  position  tests 
GCM  Position  1 biases 


1 

1 

.1 

"1 

i 

|_ 

L_ 

1 

1 

i 

□ 

□ 

9-55  X Gypto  Torquer  Scale  Factor  (KTX) 

Y Gypto  Torquer  Scale  Factor  (KTY)  -+J. 
Z Gypto  Torquer  Scale  Factor  (KTZ)  ■+[. 


arc  sec/pulse 
arc  sec/pulse 
arc  sec/pulse 


End  ETI  reading 


hr 
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FAILURE  DETECTION  AIDS  FOR  HUMAN  OPERATOR  DECISIONS 
IN  A PRECISION  INERTIAL  NAVIGATION  SYSTEM  COMPLEX 


Thomas  H . Kerr 

Member  of  the  Technical  Staff 
The  Analytic  Sciences  Corporation 
Reading,  Massachusetts 


An  existing  redundant  standby  system 
consisting  of  two  identical  Inertial  Navigation 
Systems  (INS)  may  be  improved  by  using  a third  INS 
of  higher  precision  as  a monitor.  The  monitor  INS 
provides  internal  reset  corrections  which  allow  the 
time  between  external  reference  position  fixes  to  be 
extended.  Conventional  failure  detection  aids  have 
evolved  to  enable  a human  operator  to  select  the  best 
standby  INS  for  navigation.  A new  method  for  detect- 
ing failures  in  the  monitoring  INS  has  been  developed 
and  its  performance  has  been  evaluated.  A 54  dis- 
crete state  Markov  probability  model  of  the  total 
three-INS  complex  as  a function  of  the  respective 
Mean-Time-To-Failure  and  Mean-Repair-Times  is  pre- 
sented in  this  paper.  A technique  is  included  for 
identifying  and  modeling  the  human  operator's  deci- 
sions as  non-ideal  switches,  with/without  the  new 
method  of  failure  detection  for  the  monitoring  INS. 
This  Markov  probability  model  is  used  as  an  inter- 
mediate step  in  exflicitly  specifying  how  the  new 
failure  detection  method  should  interface  with  the 
total  three-INS  complex.  This  is  accomplished  by 
quantitatively  evaluating  the  total  system  avail- 
ability. Then  the  policy  which  results  in  the 
greatest  system  availability  is  selected  as  the  most 
desirable. 
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1. 


INTRODUCTION 


f 

j 

1 


A simplified  block  diagram  of  a navigation  system, 
consisting  of  three  Inertial  Navigation  Systems  (INS),  is 
portrayed  in  Fig.  1.  This  configuration  represents  an  improve- 
ment to  the  previously  developed  standby  redundant  INS1/INS1' 
navigation  system  through  the  use  of  the  higher  precision  INS2 
as  a monitor  to  enable  internal  reset  corrections  between 
external  position  fixes.  Although  the  INS  contains  precision 
gyros,  they  incur  sufficient  drift  to  necessitate  a periodic 
corrective  action  consisting  of  an  external  position  fix.  Use 
of  the  higher  precision  INS2  for  internal  corrections  allows 
a longer  time  between  external  fixes  to  maintain  the  same  or 
better  total  system  navigation  accuracy.  The  monitoring  INS2 
is  assumed  to  only  have  the  higher  accuracy  needed  to  supply 
corrective  resets  to  INS1  gyro  drift-rates  and  not  to  have 
full  navigation  capabilities.  A more  detailed  view  of  the 
operation  and  sampling  times  of  this  three  INS  complex  is 
afforded  in  Fig.  2 where  both  the  Inertial  Navigation  Unit  and 
its  associated  controlling  software  consisting  of  Kalman  filters 
based  on  linear  error  models  of  the  component  gyros  are  shown. 
(The  use  of  Kalman  filters  in  this  role  is  standard  as  dis- 
cussed in  Ref.  1.) 


An  operator  (navigator)  oversees  the  operation  of 
this  three  INS  complex.  This  human  operator  makes  the  follow- 
ing decisions  which  are  respectively  represented  by  switches 
S'  and  S of  Figs.  1 and  2: 


operator  selects  the  mode  of  operation 
(either  prime  mode  or  back-up  mode) 

operator  selects  the  master  on-line 


INS1 . 


i 
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POSITION  FIXES 


POSITION 

FIX 

SUS SYSTEM 
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INTERNAL 
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ON-LINE  INS 
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POSITION  FIXES 
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Figure  1 Simplified  Block  Diagram  Overview  of  the 
Three  INS  Complex 


SAMPLING  AT  T . » T , > T , 
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POSITION 

REFERENCE 
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MODELING 
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t* 

RESET  1 

corrections! 


INS  I 

LOCAL-LEVEL 


FILTER 
MODELING 
INS  1 


filter  i 
modeling  |-* 

INS  1 ON^Y  } 


RESET  I 
CORRECTIONS' 

I 


I n 

I FILTER  I 
* (S-4--J  MODELING  L^._ 

\ l INS  1 ONLY  • 


INS  V 

LOCAL  LEVEL 


• SV/ITCHES  S.  S*  REPRESENT  HUMAN  OPERATOR  DECISIONS 

■ - USUAL  OPERATIONAL  MODE 

BACKUP  MODE 

ALL  S'  SWITCHES  ACTIVATED  SIMULTANEOUSLY 


NAVIGATION 

OUTPUT 


Figure  2 More  Detailed  View  of  the  INS1/INS2 
System 


The  prime  mode  uses  the  newly  configured  INS1/INS2  while 
the  back-up  mode  (represented  by  the  dashed  lines  in  Figs. 

1 and  2)  uses  only  the  standard  INS1/INS1'.  However,  both 
modes  require  the  selection  of  the  master  on-line  INS1 . 

The  operator’s  selection  of  the  master  INS1  is  based  on  the 
following  information  on  each  INS1  as  obtained  from  a computer 
or  technician's  graph: 

• plots  of  the  divergence  of  INS1 
(INS1’)  from  external  fixes 

• plots  of  INS1  (INS1')  gyro  bias 
correction  histories 

• plots  of  INS1  (INS11)  comparisons 
to  any  available  reference. 

After  the  total  problem  of  implementation  had  been 
partitioned  into  tractable  sub-problems,  our  primary  objective 
was  to  specify  a method  for  detecting  failures  in  the  INS2 . 

It  is  critical  to  detect  any  INS2  failures,  otherwise,  faulty 
internal  reset  corrections  are  provided  to  both  INS1  and  INS1'. 
Once  the  method  for  detecting  failures  in  the  INS2  had  been 
specified  and  the  performance  had  been  evaluated,  a natural 
consequence  of  the  completion  of  the  primary  objective  is 
encountered  as  the  secondary  objective  of  specifying  how  the 
INS2  failure  detector  will  interface  with  the  entire  system.  The 
interface  should  also  include  any  additional  INS2  failure  de- 
tection aids  that  are  subsequently  developed  to  help  the  naviga- 
tor decide  in  what  mode  he  should  operate  the  system. 

The  underlying  framework  of  an  analysis  of  the  three 
INS  system  of  Figs.  1 and  2 is  presented  herein  in  terms  of 
the  simplified  representation  of  Fig.  3,  using  the  standard 
reliability  theory  techniques  of  Ref.  2.  The  use  of  more 
realistic  non-ideal  switches  to  represent  the  human  operator's 
function  gives  rise  to  54  discrete  states  of  a Markov 
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Figure  3 Block  Diagram  of  Three  INS  Complex  as  Used 
in  the  Availability  Analysis 


probability  model  of  the  system  as  is  discussed  in  Section  3. 

A technique  is  proposed  for  relatively  weighting  the  failure 
monitoring  test  results  of  the  specified  INS2  failure  detection 
method  with  those  of  any  other  INS2  failure  detection  aid 
(manual  or  automatic)  that  is  subsequently  developed.  A 
quantification  of  the  total  system  availability  (i.e.,  reli- 
ability with  repair)  is  then  obtained  and  the  best  interfac- 
ing policy  for  integrating  the  proposed  INS2  failure  detector 
into  the  total  system  is  the  policy  (corresponding  to  a weight- 
ing) that  yields  the  greatest  system  availability. 


2. 


AN  I NS 2 FAILURE  DETECTION  METHOD 


I 


A summarizing  overview  is  given  in  Fig.  4 of  how 
this  failure  detection  approach  works.  The  theoretical 
basis  of  the  so-called  CR2  failure  detection  approach  is 
a generalization  of  the  use  of  confidence  intervals.  The 
three  main  ideas  that  serve  as  the  foundation  for  CR2  failure 
detection  are  shown  as  three  different  diagrams  in  Fig.  4 
and  are  discussed  below.  These  diagrams  are  shown  in  juxta- 
position to  facilitate  a comparison  of  how  the  relative 
overlapping  of  the  confidence  regions  affects  the  scalar 
test  statistic  at  three  specific  check  times,  t^,  tg,  and 
tg.  These  confidence  regions  are  portrayed  in  the  top  dia- 
gram of  Fig.  4.  At  each  check  time,  these  confidence  re- 
gions are  elliptical.  A failure  is  declared  when  the  two 
confidence  regions  do  not  overlap. 


At  each  check  time,  t^,  the  two  elliptical  cross 
sections  of  the  confidence  regions,  shown  in  the  top  diagram 
of  Fig.  4,  are  fixed  levels  of  two  paraboloids,  shown  in 
the  middle  diagram.  The  problem  is  to  determine  whether 
these  two  ellipses  overlap.  In  developing  the  real-time 
detection  algorithm,  the  test  for  the  presence  or  absence 
of  overlap  was  formulated  as  the  solution  of  a minimization 
problem.  The  relative  position  of  Jt(t^),  the  minimum  point 
of  the  intersection  of  the  two  paraboloids,  to  K^(t^),  the 
level  that  corresponds  to  the  elliptical  cross  section  of 
the  confidence  regions,  determines  if  there  is  overlap  and. 


*CR2  is  an  acronym  for  Two  (2)  Confidence  Regions. 
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Figure  4 Overview  Shows  How  Comparison  of  Scalar  Test  Statistic  to  Decision 
Threshold  Relates  to  Test  for  Confidence  Region  Overlap  Through 
Artifice  of  Optimization  Problem 


if  so,  the  amount  of  overlap.  As  long  as  £. ( t . ) is  below 
KjC^)  there  is  overlap,  but  when  i-Ct.^)  exceeds  K-^t^),  then 
the  confidence  regions  are  disjoint  and  a failure  is  declared. 
The  relationship  between  the  test  statistic  £(t^)  and  the 
decision  threshold  K^(t^)  is  summarized  in  the  bottom  diagram 
of  Fig.  4.  It  is  sufficient  to  observe  only  the  test  statistic 
&(tj),  and  to  declare  failures  when  Jl(t-i)  exceeds  K-j(  t-j  ) . 

In  the  CR2  failure  detector,  a higher  level  of  the 
threshold  K-^ , to  which  the  test  statistic  ^(t.^)  is  compared, 
effectively  raises  the  heights  of  the  ellipses  in  the  asso- 
ciated optimization  problem;  this  corresponds  to  stouter  con- 
fidence regions.  Analytic  expressions  have  been  derived  which 
are  used  for  pre-specif ying  the  time-varying  decision  threshold 
and  for  evaluating  the  instantaneous  probabilities  of  the 
test  statistic  exceeding  the  threshold  under  HQ  (no-failure) 
and  (a  particular  magnitude  failure),  respectively,  as 
Pfa  and  Pfj.  The  expressions  are  used  in  the  setting  of 
the  threshold  in  a characteristic  trade-off  of  instantaneous 

probability  of  fa?.se  alarm  P_  versus  the  probability  of  cor- 

* 1 a 

rect  detection  P^  associated  with  hypothesis  testing  detection 
decisions.  Early  derivations  of  the  CR2  failure  detection 
approach  may  be  found  in  Refs.  3 and  4,  while  the  most  recent 
theoretical  refinements  and  convergence  and  convergence-rate 
proofs  are  in  Ref.  5.  The  mathematical  techniques  of  confidence 
regions  used  in  these  derivations  are  similar  to  those  used 
in  Ref . 6 . 


♦The  probability  of  miss  is  P =1-P^. 
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Unlike  the  situations  where  the  usual  likelihood 
ratio  is  a Uniformly  Most  Powerful  (UMP)  test  in  that  it  is 
as  good  as  or  better  than  any  other  decision  test,  an  INS2 
failure  represents  a random  event  that  occurs  at  a random  or 
unknown  time.  There  is  little  special  justification  for  using  a 
likelihood  ratio  as  a decision  function  (Ref.  7,  p.  96;  Ref. 

8,  p.  315)  since  it  is  not  UMP  for  this  situation  and  there 
may  be  other  decision  functions  that  are  as  good  or  better. 
Confidence  region  tests  serve  as  one  alternative.  Recently, 
confidence  region  approaches  have  been  developed  for  other  de- 
tection applications  as  well  (Ref.  9). 

When  external  position  fixes  are  available  to  the 
three-INS  complex,  failures  in  the  individual  INS  are  more 
easily  detected  by  a comparison  to  this  more  accurate  external 
fix  as  a standard.  The  CR2  failure  detector  is  used  to  detect 
failures  (of  a certain  critical  magnitude  corresponding  to 
a certain  critical  Signal-to-Noise  Ratio  associated  with  the 
resulting  failure  signal  response)  between  external  fixes. 

The  CR2  failure  detection  performance  on  real  INS2 
data  is  presented  in  Fig.  5.  The  plot  on  the  left  of  Fig.  5 
represents  the  CR2  test  statistic  and  the  pre-specif ied  deci- 
sion threshold  under  a no-failure  condition.  After  an 
initial  24  hrs,*  the  test  statistic  is  well  below  the  deci- 
sion threshold,  confirming  that  no  failure  is  present.  The 
plot  on  the  right  of  Fig.  5 represents  the  CR2  test  statistic 
and  decision  threshold  for  a large  magnitude  ramp  drift-rate 
failure  in  one  of  the  INS2  gyros.  Following  the  initial  24 
hr  waiting  period,  the  CR2  test  statistic  quickly  exceeds 
the  threshold  correctly  indicating  a failure. 


♦The  error  dynamics  of  a gyro  reflect  the  24  hr  earth  rotation 
rate;  consequently,  approximately  one  full  24  hr  period  is  re- 
quired for  the  filter  to  lock  onto  the  cycle  of  the  underlying 
sinusoid. 


Figure  5 INS2  (CR2)  Failure  Detection 
Using  Real  Data 


3. 


AVAILABILITY  ANALYSIS  OF  THE  THREE  INS  COMPLEX 


3.1  THE  MARKOV  MODEL  FOR  THE  DISCRETE  STATE  PROBABILITIES 

The  configuration  of  Fig.  3 is  now  analyzed  using 
the  standard  reliability  theory  techniques  and  failure/repair 
models  of  Ref.  2.  These  standard  failure  and  repair  models 
are  illustrated  in  Fig.  6.  A representative  failure  density 
and  its  normalization  as  the  more  useful  "bathtub"  hazard 
rate  are  depicted  in  the  upper  two  diagrams  of  Fig.  6.  For 
each  INS,  it  is  assumed  that  operation  is  between  the  times 
t i and  t2  and  that  here  the  hazard  rate  is  a constant*  (con- 
sistent with  the  usual  model  for  these  modular  components  as 
an  exponential  failure  density  with  specified  Mean-Time-To- 
Failure) . The  lower  diagram  in  Fig.  6 is  the  standard  Erlang 
or  gamma  distributed  repair  model.  Here  again  the  official 
Navy  model  is  an  exponential  repair  model  with  specified 
Mean-Repair-Time . 


In  performing  the  availability  analysis,  the  fol- 
lowing notation  and  conventions  will  be  used. 

Modular  INS  components  have  two  states: 

Xi  = component  i is  navigating  within  specifications 

X^  ^ component  i is  not  navigating  within  specifications. 


*A  constant  hazard  rate  is  reasonable  assuming  that  infant 
mortality  failures  have  been  weeded  out  through  initial 
acceptance  testing  and  that  wear-out  failures  are  avoided 
through  periodic  preventive  maintenance. 
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The  non-ideal  switches  representing  the  human  operator 
decisions  have  three  states: 

S k switch  S is  performing  perfectly 

S k switch  S does  not  respond  to  a failed  INS1 
(i.e. , = l_Pd  < ^ ) 

S k switch  S actuates  by  mistake  while  INS1  is  good 
(i.e. , Pfa  > 0)  . 

Similarly* 


S’ 

A 

switch 

S' 

is  performing  perfectly 

S' 

A 

switch 

S' 

does  not  respond  to 

a failed  INS2 

(i.e. , 

P ' 
M 

Sl~Pd 

I' 

A 

switch 

S' 

actuates  by  mistake 

while  INS2  is  good 

(i.e. , 

Pfa  > °>  * 

The  number  of  discernable  discrete  states  in  a totally 

exhaustive  enumeration  of  the  possible  combinations  of  failed 

and  unfailed  INS  and  satisf actory/f alse-alarming/missing 

3 2 

switches  in  the  5-element  model  of  Fig.  3 is  (2)  (3)  = 72. 

Since  Xg  (INS1)  is  identical  to  (INS1')  as  a warm  stand-by 
system,  the  total  number  of  states  may  be  collapsed  to  54 
as  indicated  in  Ref.  2 and  explicitly  enumerated  in  Table  1. 

The  following  further  superscript  notation  is  used  in  Table  1: 

t k Total  System  Failure.  No  good  navigation  informa- 
tion is  available. 

* k Partial  System  Failure.  Only  good  INS1  or  INS1 ' 
navigation  information  is  available. 


♦This  reliability  analysis  is  applicable  to  any  INS2  failure 
detection  method  that  uses  the  same  filter  inputs  as  used 
by  the  CR2  test  as  long  as  the  P'  and  PL  which  characterize 
it  have  been  evaluated.  a 
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TABLE  1 

AN  EXHAUSTIVE  ENUMERATION  OF  THE  DISCRETE  STATES 
IN  THE  STANDBY/SERIES  5 ELEMENT  SYSTEM  OF  FIG.  3 


S1  'S'W3S 

Si9=S'Xix2X3S 

S37'SX1X2>‘3S 

S2  “S'X1X2X3S 

S20=S’X1X2X3§* 

S38“S  X1X2X3S 

s3  «s,x1x2x3s+s,x1x2x3s 

S21=S'X1X2X3§tS'X1X2X35' 

S39=§,XlX2X35+I'XlX2X33i 

s4  -S'X1x2x3s+s-X1x2X3^ 

S22“S ' X1X2X3S+S ' X1X2X3S 

S40=f'XlX2X3S'+S'XlX2X3§’ 

S5  =S'X1X2X3S+ 

S23=S'W3§+ 

S41=§'XlW+ 

S6  =S' W3S' 

S24=S’XlX2X3St 

S42“S,X1X2X3S+ 

S7  -S'XjX^S 

S25-S'X1X2X3§ 

S43=S  X1X2X33 

Sg  -5,X1X2X3S+ 

S26'S'XlX2X3g' 

S44°^  XlX2X3g+ 

S9  -5-X1J’2X3S+S'X1X2X3S 

S27=S ' XjXgXgS+S  • XjXjX^. 

S45=g  XlX2X3g'S ' Xj^XjXjS 

S10-5'W3S+3'W3St 

S28=S'XlX2X33>S'XlX2X3gt 

S46=3'XlX2X3g+S'XlX2X3§t 

Sll”5’Xi*2*3S+ 

S29=S'XlX2X3f+ 

S47'S'Xl¥3!t 

S12-S'W3S+ 

s3o-s’W3r 

S48=S'X1X2X3r 

S13“§’X1X2X3S* 

S31*S'X1X2X35 

S49=f'XlX2X3r 

S14-I,X1X2X3S 

s32-s-x1x2x3r 

S50=f'XlX2X3§* 

S15=§'XlX2X3S+f'XlX2X3S‘ 

S33=S,X1X2X3§+S'X1X2X3^ 

S51=S'X1X2X33+5'X1X2X3§+ 

S16-5'X1X2X3S^'X1X2X3S* 

S34=1'XlX2X35T5'XlX2X3r 

S52=f’XlX2X3S+f,XlX2X3§t 

S17*S'W3st 

S35-g’XlX2X3St 

S18'S’X1X2X3S+ 

S36“S'XlX2X3gt 

S54=I’XlX2X3r 

* = Partial  System  Failure 
t = Total  System  Failure 


It  may  be  summarized  from  Table  1 that  there  are 

• 9 states  for  good  INS1/INS2  navigation 
information 

• 11  states  for  good  INS1  (or  INS1') 
navigation  information  only 

♦ 

• 34  states  for  having  no  good  navi- 
gation information 

for  a total  of  54  discrete  states. 


*It  is  considered  that  no  good  navigation  information  is 
available  when  both  INS1  and  INS1'  are  failed  even  though 
the  INS2  may  be  unfailed  because  the  INS2  is  assumed  to  only 
have  higher  precision  gyros  and  not  to  have  full  navigation 
capabilities.  Therefore,  as  indicated  in  Fig.  3,  it  may  not 
be  used  alone  for  navigation. 


Associated  with  this  5-element  system  is  a vector 
of  probabilities  corresponding  to  the  likelihood  that  the 
system  of  Fig.  3 will  be  found  in  a particular  state  of  the 
54  discrete  states  enumerated  in  Table  1 at  a specified  time. 
In  general,  the  probabilities  of  occupying  a particular  state 
change  as  a function  of  time  and  the  specific  manner  of  inter- 
connection and  satisfy  a linear  difference  equation  of  the 
following  form: 


rp„  ( t+A >i 
bi 

rps  (^1 
bl 

Ps  (t+A) 

Po  (t) 
b2 

• 

= 

TRANS I 

TION 

• 

• 

PROBABILITIES 

• 

Pq  (t+A) 

Ps  (t) 

b54 

U _ 

_ 

b54 

This  equation  characterizes  the  time  evolution  of  the  abso- 
lute probability  of  each  of  the  54  possible  states  being 
assumed  by  the  system  as  determined  by  the  transition 
probabilities.  An  exhaustive  enumeration  of  the  54 
states  in  Table  1 which  the  system  of  Fig.  3 can  assume  is 
needed  as  a necessary  intermediate  step  in  specifying  the 
elements  of  the  associated  transition  probability  matrix,  T 
in  Eq . (1),  that  characterizes  the  specific  system  inter- 
connections and  manner  of  use. 

Two  representative  examples  are  presented  here 
to  demonstrate  how  the  elements  of  the  transition  probability 
matrix  are  determined.  Consider  the  following  state  as  the 
focus  of  attention 
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S16  = S'XjXjXgS+S'XjXjXjS 


(2) 


which  represents  switch  S'  in  a mode  where  it  would  be  prone 
to  false  alarm  (whether  X.^  is  failed  or  not),  but  with 
(INS2)  in  a condition  of  failure,  and  either  X^CINSI)  or 
Xg(INSl')  in  a failed  condition  (but  not  both).  The  state 
S16  1S  one  designated  as  a partial  systems  failure  because 
there  is  one  INS1  available  and  switch  S is  neither  in  the 
false  alarm  mode  nor  in  the  miss  mode  so  the  selection  of 
the  proper  INS1  (INS11)  to  use  in  navigation  is  correctly 
made  and  good  navigation  information  is  available  but  only 
from  an  INS1  (INS1' ) . 

In  the  flow  diagram  of  Fig.  7,*  the  state  S16  may  be 
entered  at  time  = t+A  (notation:  Slg(t+A))  only  through  the 

paths  shown  from  the  five  states  S . , S. „ , S1 c , S.„,  S, 0 if 
they  were  occupied  at  time  = t‘  (notation:  S4(t),  S^^t), 

etc.).  The  transition  probability,  t^,  of  going  from  one 
state  S.  to  a distinctly  different  state  S.  is  the  feed- 
forward  hazard  rate  times  the  time  step  A,  e.g.,  the  transi- 
tion probability  of  going  from  state  S14(t)  to  Slg(t+A)  is 


t16 , 14 


2A^  • A 


(3) 


The  probability  of  occupying  the  same  state  at  time  = t+A, 
S^(t+A),  as  was  occupied  on  the  previous  step  at  time  = t, 
Si(t),  is* 


♦Reciprocals  of  MTTF  and  Mean-Repair-Time  for  the  INS1  and 
INS2  are,  respectively,  A^,  y^  and  A Pg- 

tThis  model  uses  the  standard  reliability  assumption  of  only 
one  allowable  change  in  status  during  one  transition  step=A . 

+-These  are  elements  along  the  principal  diagonal  of  the 
transition  probability  matrix,  T. 
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Figure  7 


Allowable  States  at  Time=t  That  May  Enter 
State  S^g  at  Time=t+A 


■f* 

t.  = 1 — T ( j hazard  rate)  • A 
1,1  u 

j ranges  over  all  feedforward 
paths  for  switches  and  failures 
connecting  with  state  S^.j^i 


e.g.,  the  transition  probability  of  being  in  state  S^g  a 
time  = t+A  given  that  state  S1C  was  occupied  at  time  - 


t 


From  Fig.  8,  the  state  S^g  at  time  = t is  an  allowable  entry 

state  for  states  S14,  s15 »si6 ’ S18 ’ S40 ’ S52  at  time  = t+A- 
Therefore,  the  column  of  the  transition  probability  matrix 
T,  to  be  multiplied  by  Pg4g(t)  in  Eq.  (1),  has  non-zero 
path  entries  only  on  the  rows  corresponding  respectively  to 

S14’S15,S16’S18’S40’S52‘  This  same  type  of  analysis  is 
performed  for  each  of  the  other  54  states  in  order  to 
completely  specify  the  transition  probability  matrix  T.* 

As  indicated  by  the  above  representative  example 

of  state  S^g,  T is  a rather  sparse  matrix.  Also,  notice  that 

the  elements  of  T are  the  constants  P.,P.  ,P',P'  , A.. , A0 , p, , y_ ; 

d fa  d fa  1 2 1 2 

this  result  is  a consequence  of  the  exponential  repair  models 
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and  the  constant  hazard  functions  of  the  exponential  failure 
models.  Since  the  transition  probabilities  are  all  independent 
of  time  and  depend  only  on  A,  the  underlying  Markov  process  is 
termed  homogeneous.  Since  the  homogeneous  process  of  this 
analysis  does  not  have  the  property  that  any  state  may  be 
reached  directly  from  any  other  state  with  positive  probability 
at  each  time,  this  system  is  not  ergodic  (Ref.  2).  The  vector 
iteration  equation  of  Eq.  (1)  is  linear  and  time  invariant  for 
the  system  of  this  analysis  and  is  therefore  extremely  con- 
venient to  work  with. 


3.2  FORMULATING  TWO  APPROPRIATE  AVAILABILITY  EXPRESSIONS 

As  indicated  in  Section  3.1  and  in  Table  1 by  * 
and  + there  are  two  types  of  good  navigation  information 
that  are  available  from  this  three-INS  complex.  When  the  three 
INS  complex  is  operating  in  a well-functioning  state  repre- 
senting the  primary  mode,  there  is  high  quality  INS1/INS2 
navigation  information.  When  the  three-INS  complex  is 
operating  in  a well-functioning  state  representing  the  back-up 
mode,  there  is  lower  quality  INS1/INS1'  navigation  information 
(with  more  frequent  external  reference  position  fixes  required) 
To  reflect  the  two  different,  but  successful,  operational 
modes  of  the  system,  the  former  being  more  desirable  as  the 
principal  objective  than  the  latter,  two  different  availability 
functions  are  defined.  Availability  functions  which  are,  by 
definition,  the  probability  of  operating  successfully  with 
repair  (in  contradistinction  to  reliability  which  consists 
of  the  same  probabilities  but  without  allowing  the  policy  of 
repair)  were  chosen  to  correspond  to  the  situations  of 

(1)  having  only  successful  INS1/INS2 
navigation  information 

(2)  having  any  successful  navigation 
information,  either  INS1/INS2  or 
just  INS1/INS1 ' . 
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These  two  availability  expressions  corresponding,  respectively, 
to  1 and  2 above,  are 


Al/2(t)  = PS  (t>  + ps  (t)  + pq  (t)  + Po  (t)  + P„  (t) 
bl  o S7  S9  S19 


+ Ps  (t)  + Pq  (t)  + Pg  (t)  + P_  (t)  (7) 

^ o o & a n 


'25 


31 


38 


43 


A1/r(t)  k A1/2(t)  + Pg  (t)  + Pg  (t)  + Pg  (t)  + Pc  (t) 


'13 


'14 


+ P (t)  + P (t)  + P (t)  + P„  (t) 
b15  S16  S20  S26 


+ P_  (t)  + P_  (t)  + Pc  (t) 


'37 


'49 


50 


(8) 


where  the  PSi(t)  are  the  unconditional  probabilities  of  being 
in  the  states  at  time  t,  with  the  evolution  being  completely 
specified  by  the  iteration  equation  of  Eq . (1).  Notice  that 
Af/i',  the  availability  of  any  good  navigation  information  what- 
soever, wholly  contains  A1^2(t),  the  availability  of  the 
INS1/INS2,  in  addition  to  the  non-negative  quantities  repre- 
senting the  unconditional  probabilities  of  being  in  the 
states  designated  as  partial  failures. 


3.3  A PROPOSED  NEW  MODEL  FOR  THE  HUMAN  OPERATOR  DECISION 

It  is  standard  practice  to  model  the  decisions  of 
a human  operator  as  a switch  (Ref.  2).  When  a more  detailed 
and  accurate  analysis  is  warranted,  it  is  also  standard  to 
acknowledge  that  the  switch  is  not  really  perfect  (Ref.  10) 
but  inherently  contains  a detector  which  may  sometimes  miss 
(i.e.,  switch  remains  in  position  because  of  a failure  to 
recognize  that  conditions  are  such  that  it  should  have  been 
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thrown)  and  false  alarm  (i.e.,  switch  has  flipped  even  though 
conditions  do  not  warrant  it).  The  ensemble  statistics  of 
the  imperfect  switch  which  completely  characterize  its  behavior 
are  its  inherent  probability  of  false  alarm  and  its  probability 
of  miss. 


Switch  S represents  the  human  operator's  decision 
in  the  selection  of  the  master  INS1,  as  aided  by  the  infor- 
mation from  the  accessible  computer  outputs  and  plots 
mentioned  in  Section  1.  Switch  S'  represents  the  human  opera- 
tor's decision  in  the  selection  of  the  operational  mode.  To 
date,  the  only  INS2  failure  detection  technique  available  for 
use  as  the  detection  element  (between  external  position  re- 
sets) is  of  the  form  of  the  CR2  technique  and  consequently, 
its  Pd  and  Pf  characteristics  apply  directly  to  switch  S' 
only  if  it  is  used  exclusively.  For  the  case  where  the  CR2 
technique  is  used  exclusively,  the  effective  characterizing 
ensemble  statistics  for  switch  S'  are: 


Pfa’eff 

OP 

= P^  , CR2 
fa’ 

(9) 

P',eff 

= P°P , CR2 

(10) 

where  P^a,eff  (P^.eff)  represents  the  effective  Pfa(Pd)  of 
switch  S’  and  P^a,CR2  (P£j,CR2)  represents  the  particular 
characterizing  statistic  of  the  CR2  failure  detector  with 
the  Operating  Point  (OP)  being  determined  by  the  level  of 
the  pre-specif ied  decision  threshold  that  is  used. 

However,  other  INS2  failure  detection  techniques  (possibly 
manual)  are  currently  under  development  and  upon  completion 
may  also  be  used  in  the  detection  element  of  switch  S’.  If 
these  new  techniques  are  used  exclusively,  then  the  effective 
characterizing  ensemble  statistics  for  switch  S'  are 
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P'  ,eff  = P.  .manual 
1 2-  I a 


(id 


OP ' 

P^.eff  = P~j  .manual  (12) 

where  the  subscript  "manual"  is  just  used  to  distinguish 
these  characteristics  from  those  associated  with  the  CR2 
technique . 


A natural  question  to  ask  is  "can  the  two  detection 

techniques  be  used  together  to  enhance  performance  and  if  so, 

what  would  be  an  optimum  mix?"  Any  attempt  to  answer  this 

question  requires  a model  for  the  effective  P,  and  P„  of 

d fa 

switch  S'  when  both  failure  detection  methods  are  used.  The 
new  model  that  is  being  proposed  to  effectively  characterize 
switch  S'  when  both  failure  detection  techniques  are  being 
used  simultaneously  with  relative  weightings  (1-X)  and  X is 

op  ' OP 

Pja, ef f = (1-X) -Pja, manual  + X-Pja,CR2  (13) 

P^.eff  = (1-X) -P°P 'manual  + X-P°P,CR2  (14) 

where 

0 < X < 1 (15) 

Notice  that  this  is  a linear  interpolative  model  which  incor- 
porates the  following  highly  desirable  properties: 

• For  X=l,  the  proposed  model  of  Eqs.  (13), 

(14)  reduces  to  the  standard  result  of 
Eq.  (9), (10). 

• For  X —0 , the  proposed  model  of  Eqs.  (13), 

(14)  reduces  to  the  standard  result  of 
Eq.  (11), (12). 

• For  any  X such  that  Eq . (15)  is  satisfied, 
the  proposed  model  of  Eqs.  (13),  (14)  re- 
sults in  quantities  that  have  all  the  requi- 
site properties  to  be  probabilities  (i.e., 


-v  fcK-WjgMK, 
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0 < PL  , ef f < 1 
— fa  — 

(16) 

0 < P’ ,eff  < 1 
— d ’ — 

(17) 

as  a natural  consequence  of  the  same 
inequality  being  satisfied  by  , manual; 
Pd,CR2;  Pfa,manual;  and  Pfa>CR2). 

• The  combined  use  of  both  failure  de- 
tection techniques  corresponding  to  this 
proposed  linear  model  may  be  implemented 
easily  by  using  the  linear  weightings 
in  the  combined  decision  rule  as  dis- 
cussed below. 


A combined  decision  rule  corresponding  to  the  above 
proposed  model  is  easily  obtained.  The  combined  decision 
rule  is  as  follows: 


(1)  if 

(l-X)-£2  + X-£1  < (1-X)-K2  + X-K1 


then  choose  H^:  no-failure 

(2)  if 

(l-X)-£2  + X-£1  > (1-X)-K2  + \-K1 


(18) 


(19) 


then  choose  H^:  a failure  (having  the  critical  SNR  magnitude) 

where  is  the  test  statistic  and  is  the  pre-specif ied 
decision  threshold  of  the  CR2  failure  detection  technique 
while  Jt2  is  the  test  statistic  and  K2  is  the  pre-specif  ied 
decision  threshold  of  the  so-called  manual  failure  detection 
technique.  Notice  that  for  the  two  extremes  of  X=0  or  X=1 , 
the  decision  rule  of  Eqs.  (18)  and  (19)  reduces  to  the  usual 
decision  rules  for  each  separate  detection  technique. 
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This  proposed  model  offers  an  additional  appeal 
when  the  so-called  Receiver  Operating  Characteristics  (ROC) 
of  Fig.  9,  which  under ly  all  binary  decision  tests  (Ref.  7), 
are  considered.  The  two  hypothetical  solid  curves  in  Fig.  9 
represent  the  complete  operating  capability  ranges  of  the  two 
failure  detection  techniques.  For  each  of  the  failure 
detectors,  the  usual  practice  is  to  trade-off  P^  versus  Pfa 
by  deciding  upon  a compromise  operating  point  (usually  chosen 
to  be  in  the  vicinity  of  the  "knee"  of  the  curve) . This 
operating  point  is  fixed  by  fixing  the  decision  threshold. 

The  operating  points  of  the  two  curves  are  represented  by 
OP  and  OP'  and  are  denoted  by  the  corresponding  two  coordinates 
(Pfa.Pd)-  In  general,  the  plane  cannot  be  ranked  (only  the 
real  line  R may  be  ordered);  so,  in  general,  it  is  impos- 
sible to  say,  unequivocally,  that  one  operating  point  is  better 
than  another.  In  Fig.  9,  OP  has  a desirably  lower  Pfa  while 
OP'  has  a desirably  higher  P^.  Use  of  the  proposed  model  of 
Eqs.  (13),  (14),  (15)  would  result  in  an  effective  OP"  somewhere 
on  the  dotted  line  between  OP  and  OP' . For  X=0.5,  the  effective 
OP"would  be  the  mid-point  of  this  line  segment.  The  selection 
of  the  preferred  weighting  X and  consequently  the  associated 
operation  point  is  discussed  in  Section  3.5. 


Figure  9 Representative  ROC  Underlying  all  Binary 
Detection  Tests 
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3.4  USE  OF  OPERATOR  TO  INTERPRET  RESULTS  OF  FAILURE  DETECTION 
TESTS 

Even  though  the  decision  rules  of  the  INS2  failure 
detector,  as  presented  in  Section  3.3,  are  instantaneous 
rules,  it  is  a common  practice  to  allow  a human  operator 
to  intervene  in  the  interpretation  of  the  outcome  of  the 
test  before  a switch  is  thrown.  This  procedure  is  followed 
for  conservatism  to  exploit  the  useful  adaptive  capability 
of  a human  operator  to  pick  out  strong  trends  over  a time 
interval  and,  in  effect,  to  create  a test  with  memory, 
rather  than  to  let  an  automatic  instantaneous  computerized 
test,  which  could  be  subject  to  spurious  noise  transients, 
control  the  throwing  of  a switch.  Use  of  the  human  operator 
in  this  way  would  change  the  effective  P^  and  P^a  character- 
istics of  switch  S'  from  what  would  be  dictated  by  con- 
sidering only  the  instantaneous  P^  and  Pja  characteristics 
of  the  CR2  INS2  failure  detector.  However,  once  the  effective 
P(j  and  Pja  of  the  human  operator  interpreting  the  results  of 
the  CR2  test  over  a time  interval  are  evaluated,  they  may  be 
substituted  for  P^,  CR2  and  Pfa,  CR2  in  the  availability 
analysis  of  Sections  3.1  and  3.2. 

More  realistic  effective  characteristic  P^  and  Pfa, 
which  are  associated  with  the  human  operator's  use  of  the 
instantaneous  CR2  INS2  failure  detector  test  results  to 
recognize  trends  over  a time  interval,  may  be  obtained  by 
solving  an  associated  level-crossing  problem  (i.e.,  the 
crossing  of  the  Gaussian-related  scalar  test  statistic  above 
the  deterministic  decision  threshold  level  over  a time 
interval).  Usually  level-crossing  problems  are  not  very 
tractable  (Ref.  11);  however,  these  interval  probability 
calculations  are  presently  being  pursued  with  an  analytically 
tractable  upper  bound  (Ref.  12)  which  can  be  optimized  to  be 
as  tight  as  possible  to  the  objective  interval  probabilities 
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through  the  vehicle  of  a standard  quadratic  programming 
problem  (Ref.  13,  14).  The  quadratic  programming  problem 
arises  naturally  in  a discrete-time  mechanization  of  the 
upper  bound. 


The  close  connection  between  level-crossing  prob- 
lems and  the  test  statistic  crossing  above  a decision  threshold 
was  also  recognized  in  Ref.  15,  where  a completely  different 
failure  detection  approach  was  investigated.  However,  only 
the  calculation  of  the  expected  level -crossing  times  were 
attempted  there. 


3.5  MECHANICS  OF  OPTIMAL  POLICY  SELECTION  BASED  ON 
ASYMPTOTIC  AVAILABILITY 

It  is  well-known  that  the  reliability  function  ; 

(without  repair)  asymptotically  goes  to  zero  with  increasing 
time  (Ref.  2).  It  is  also  well-known  that  the  availability 
function  asymptotically  goes  to  some  constant  positive  value 
as  time  increases  (Ref.  2).  Applying  this  principle  to 
Al/2(t)*  of  Eq.  (8)  with  the  switch  S'  characterized  by  the 
proposed  model  of  Eqs.  (13),  (14),  (15)  allows  the  evaluation 
of  the  asymptotic  levels  of  A^^Ct)  over  a range  of  X weightings 
between  0 and  1.  The  value  of  X that  yields  the  largest 
asymptotic  A^^t)  would  be  chosen  and  used  in  the  decision 
rule  of  Eqs.  (18),  (19)  as  the  optimal  interfacing  of  the 
two  techniques. 


*Ai/2(t)  rather  than  A]_/i(t)  was  chosen  since  it  represents 
the  principle  objective  of  operating  in  the  primary  mode. 
The  methodology  is  equally  applicable  to  Aiyi'(t)  however 
and  it  is  conceivable  that  Ai/i'(t)  would  be  used  for  the 
evaluation  when  interest  is  in  having  any  good  navigation 
information  available. 


.r  * warna 
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SUMMARY  AND  CONCLUDING  DISCUSSION 
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The  principal  objective  was  to  specify  and  evaluate 
an  INS2  failure  detection  method.  The  CR2  failure  detection 
method  was  specified  and  its  performance  was  first  evaluated 
theoretically,  then  demonstrated  through  simulations. 

Finally,  the  CR2  failure  detection  method  was  used  on  real 
system  data  to  demonstrate  that  the  algorithm  is  robust 
enough  to  handle  real  world/model  mismatches  without  false 
alarming  while  still  maintaining  the  ability  to  detect 
failures  that  occur.  The  performance  of  the  CR2  failure 
detection  method  on  real  data  was  consistent  with  the 
theoretical  predictions.  An  abridged  view  of  the  CR2 
failure  detection  technique  and  the  milestone  of  achieving 
satisfactory  performance  with  real  data  were  given  in  Section  2. 

Upon  completion  of  the  primary  objective,  it  was 
then  necessary  to  specify  how  the  CR2  INS2  failure  detector* 
should  interface  with  the  entire  system,  including  any  addi- 
tional INS2  failure  detection  aids  (manual  or  automatic)  that 
are  subsequently  developed  to  help  the  navigation  operator 
decide  in  what  mode  he  should  operate  the  system.  It  is  also 
necessary  to  evaluate  how  the  CR2  failure  detector  affects 
the  whole  three-INS  complex.  A rigorous  theoretical  frame- 
work was  detailed  in  Section  3 for  using  the  standard  tech- 
niques of  reliability  theory  to  determine  the  effect  of  the 
characteristic  an.d  Pfa  of  the  CR2  failure  detector  on  the 


♦Method  applicable  to  any  INS2  failure  detection  method  that 
uses  the  same  filter  inputs  as  used  by  the  CR2  test  as  long 
as  the  characterizing  Pj  and  Pfa  have  been  evaluated. 
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overall  asymptotic  availability  of  the  three-INS  complex. 

When  interfacing  with  another  INS2  failure  detection  method 
is  required,  the  proposed  new  model  of  the  effective  Pd  and 
Pfa  characteristics  associated  with  the  switch,  used  as  a 
standard  model  for  human  operator  decisions,  may  be  used  to 
determine  the  optimum  relative  weightings  between  the  two 
methods  that  achieves  the  highest  asymptotic  availability. 

The  same  relative  weightings  are  also  inherited  by  the  two 
INS2  failure  detection  techniques  in  a joint  implementation 
as  discussed  in  Section  3.3. 

When  only  one  INS2  failure  detector  is  used,  the 
proposed  new  model  for  effective  Pd  and  Pfa  (for  switch  S' 
in  mode  selection)  reduce  to  the  standard  model  without  con- 
troversy. Presently,  there  appears  to  be  a scarcity  of  ROC 
data  on  human  operator  decisions  on  master  INS1  selection. 
Hopefully, this  void  will  be  filled  in  the  future  through  the 
compilation  of  adequate  performance  data  and/or  statistical 
design  of  experiments  to  quantitatively  evaluate  the  effect 
of  man-in-the-loop . The  theoretical  framework  for  quantita- 
tively evaluating  the  effect  of  the  CR2  INS2  failure  detector 
has  been  completely  worked  out,  but  is  waiting  on  the  speci- 
fication of  Pd,  Pfa  characteristics  for  switch  S since  this 
switch  (operator  decision)  is  depended  upon  in  both  the  pri- 
mary and  back-up  mode  of  the  three-INS  complex  and  availability 
results  obtained  by  proceeding  without  this  critical  informa- 
tion may  be  very  misleading. 

One  methodology,  recently  proposed  for  modeling  man- 
machine  availability  as  an  allocation  problem  (Ref.  16),  allows 
several  levels  of  complexity  and  additional  realism  such  as: 
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• grade  level  and  years  of  experience 
of  each  repairman 

• number  of  repairmen  assigned  to  each 
repair  action 

• time  spent  by  each  repairman  on  the 
repair  . 

However,  dynamic  programming  is  required  to  solve  even  the 
most  simple  non-degenerate  problems  within  this  framework. 

In  spite  of  some  relatively  recent  theoretical  strides  in 
the  area  of  implementing  dynamic  programming  algorithms 
(Ref.  17,  18,  19)  to  reduce  the  so-called  "curse  of 
dimensionality"  associated  with  dynamic  programming  problems 
(Ref.  20),  the  implementation  of  a dynamic  programming 
algorithm  still  remains  a rather  formidable  problem  and 
usually  requires  a large  computer  allotment.  In  contrast, 
the  main  computation  required  for  the  availability  analysis 
of  Section  3 is  a relatively  minor  iteration  of  a sparse, 
constant  coefficient,  linear  difference  equation  of  medium 
order.  A more  detailed  model  may  be  obtained,  using  the 
technique  of  Section  3,  by  modeling  more  elements. 

Theoretical  strides  have  been  made  recently  in 
applying  the  techniques  of  modern  control  to  the  problem  of 
modeling  the  effect  of  the  operator-in-the-loop  on  the  per- 
formance of  the  overall  system.  Two  different  approaches 
which  both  make  use  of  Kalman  filters  are: 

(1)  human  operator  modeling  within  the 
theoretical  framework  of  an  optimal 
stochastic  regulator  (Ref.  21,  22,  23) 

(2)  modeling  of  human  operator  remnants 
using  maximum  likelihood  methods  for 
parameter  identification  (Ref.  24,  25). 


It  is  anticipated  that  these  new  modeling  approaches  will 
yield  more  accurate  final  evaluations  of  the  effective  P^ 
and  Pfa  of  the  human  operators. 

The  methodology  discussed  in  this  paper  may  also 
have  potential  application  to  those  USAF  situations  in  which 
a human  operator  must  decide  between  two  or  more  INS  or 
navigation  aids  such  as  in: 

• the  C-141  dual  IMU  installation 

• the  C-5  planned  retrofit 

• the  B-l  . 
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ANALYTICAL  MODELS  OF  MAITNENANCE  DECISION  PROCESSES 

by 

W.  Stephen  Demmy 
Wright  State  Univeristy 


This  paper  presents  a series  of  analytical  models  of  maintenance 
decision  processes.  In  the  context  of  a given  maintenance  system,  these 
models  should  be  useful  in  answering  questions  such  as : 

1.  What  are  the  key  maintenance  decisions? 

2.  How  can  we  improve  these  decisions? 

3.  What  is  the  expected  impact  of  a specific  change  in  one 
area  upon  the  entire  system? 

In  this  section,  we  briefly  sketch  the  characteristics  of  the 
logistics  support  systems  for  the  KT-73  Inertial  Measurement  Unit,  a 
major  component  of  the  navigation  system  for  A-7D  and  A-7E  aircraft.  As 
we  shall  see,  this  system  is  extremely  complex.  This  complexity  is  not 
unique  to  the  KT-73  logistics  system;  many  other  Air  Force  items  have 
logistics  systems  of  similar  or  greater  complexity. 

Our  ultimate  objective  is  to  develop  tools  for  optimizing  the 
decision  processes  of  such  complex  networks.  In  working  toward  this  goal. 
Section  II  reviews  the  current  state  of  the  art  for  optimizing  maintenance 
decision  processes.  Next,  we  discuss  the  features  of  a series  of  increasingly 
complex  maintenance  decision  systems.  Section  III  discusses  the  major  fac- 
tors involved  in  maintenance  decisions  for  single  tests,  while  Section  IV 
presents  methods  for  setting  optimum  cutting  scores  for  such  tests.  Section 
V generalizes  the  results  of  Section  IV  to  consider  tests  in  which  multiple 
readings  are  obtained;  for  example,  a single  test  of  an  inertial  measure- 
ment unit  may  provide  readings  three  separate  slew  rates;  one  reading  for 
each  of  the  X,  Y,  and  Z coordinates.  Methods  for  determining  optimum  multi- 
variate cutting  scores  are  presented  in  this  section. 

In  Section  VI,  networks  of  test  and  repair  facilities  are  considered, 
and  a method  for  computing  optimal  cutting  scores  for  the  system  of  tests  is 
presented. 

In  most  systems,  the  detailed  data  required  to  apply  the  tools 
developed  in  Sections  III  through  VI  will  not  be  readily  available,  but 
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may  be  obtained  using  experimental  design  and  cost  analysis  techniques 
described  in  the  main  body  of  this  report.  Applying  these  techniques 
can  be  expensive;  what  is  needed  is  a tool  for  evaluating  the  potential 
savings  that  may  be  obtained  using  these  measurement  techniques.  Hence, 
Section  VII  discusses  a procedure  for  identifying  critical  test  or  repair 
activities,  i.e.,  activities  that  offer  high  potential  paybacks  if  current 
test  and/or  repair  errors  are  reduced. 

Let  us  now  consider  the  characteristics  of  the  KT-73  logistics 
support  system. 


THE  KT-73  INERTIAL  MEASUREMENT  UNIT 


The  KT-73  Inertial  Measurement  Unit  (IMU)  is  a major  component 
of  the  navigation  system  installed  in  A7D  and  A7E  aircraft.  At  present, 
the  Aerospace  Guidance  and  Metrology  Center  (AGMC)  is  the  only  depot  re- 
pair facility  for  this  IMU. 

The  KT-73  IMU  is  basically  a gyroscopically  stabilized  "platform" 
upon  which  are  mounted  electro-mechanical  accelerometers  that  sense  and 
measure  inertia  variances  associated  with  movement  of  the  unit.  During 
flight,  the  external  case  of  the  IMU  pitches  and  rolls  with  the  aircraft 
to  which  it  is  attached;  but  inside  the  IMU,  the  gyroscopically  stabilized 
platform  remains  in  the  same  constant  spatial-attitude  regardless  of  the 
movements  of  the  external  case.  Since  this  stabilized  platform  presents 
an  essentially  "fixed"  attitude  reference,  the  accelerometers  mounted  on 
it  can  provide  accurate  data  concerning  acceleration  of  the  IMU  (and  there- 
fore the  aircraft)  along  any  vector  (direction)  from  a chosen  reference 
point.  By  comparing  these  measurements  with  elapsed  time,  the  IMU  can 
determine  distance  and  direction  information  that  provides  a basis  for 
navigation  of  the  aircraft. 

The  basic  physical  construction  of  the  KT-73  IMU  is  shown  in 
a simplified  form  in  Figure  1.  From  the  figure  it  can  be  seen  that  the 
IMU  may  be  considered  to  be  basically  composed  of  three  "nested"  assem- 
blies. The  gyroscopically  stabilized  platform  (not  visible  in  Figure  1)  is 
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located  inside  the  assembly  marked  "Gimbal  Case".  The  "Gimbal  Case"  is, 
in  turn,  mounted  inside  the  "Internal  Case";  and  finally  this  compound 
assembly  is  mounted  within  the  "External  Case".  Note:  These  are  function- 
ally representative  terms  only;  they  are  not  the  actual  terms  used  by 
AGMC  to  describe  this  assembly. 

The  compound  assembly,  mentioned  above,  consisting  of  the  "Gim- 
bal Case"  mounted  within  the  "Internal  Case"  is  termed  the  gimbal  cluster 
assembly  (GCA) . 

Inside  the  GCA,  in  addition  to  the  gyroscopes , accelerometers , 
torque  motors,  and  position  plckoffs  which  comprise  the  gyroscopi- 
cally  stabilized  platform,  there  are  also  Internal  Electronic  Modules  which 
interface  the  above  electro-mechanical  assemblies.  These  Internal  Electronic 
Modules  (hereafter  referred  to  simply  as  "internal  modules")  are  basically 
plug-in  type  printed  circuit  boards. 

Physically  similar  printed  circuit  boards  are  mounted  within  the 
"External  Case";  these  circuit  boards  are  referred  to  as  "external  modules". 

The  basic  activity  involved  in  repairing  a malfunctioning  IMU  is 
simply  the  removal  and  replacement  of  the  discovered  defective  part,  i.e., 
a torque  motor  is  removed  and  replaced,  or  an  external  module  is  removed 
and  replaced.  These  removed  components  may  in  turn  be  repaired  by  removal 
and  replacement,  by  recalibration,  or  other  repair  actions.  Due  to  the 
extreme  complexity  of  the  IMU,  each  of  these  activities  may  affect  the 
operation  of  a previously  "good"  part  or  assembly.  Consider,  for  example, 
the  removal  and  replacement  of  an  internal  module.  Since  the  internal  module 
is  located  within  the  GCA  assembly,  this  entire  assembly  must  be  removed 
from  the  external  case  and  "split  open”  to  gain  access  to  the  bad  internal 

module  (see  Figure  1).  These  actions  required  to  gain  access  to  the  elec- 
tronic circuit  may  often  disturb  the  important  physical  positioning  or 
alignment  of  sensitive  electro-mechanical  components  such  as  torque  motors 
or  gyroscopes.  Therefore,  the  removal  and  replacement  of  an  internal  module 
may  necessitate  the  physical  realignment  of  a torque  motor.  From  this  example 
it  can  be  seen  that  the  seemingly  straightforward  removal  and  replacement 
of  an  assembly  or  module  located  deep  within  the  "nested"  assemblies  will 
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necessitate  multiple  disassembling  to  gain  accessability , which  in  turn 
may  require  the  physical  realignment  of  many  electro-mechanical  components. 

On  the  other  hand,  removal  and  replacement  of  a malfunctioning 
external  module  would  not  require  the  other  internal  assemblies  to  be 
disturbed  in  an  effort  to  gain  access  to  the  defective  part.  Therefore 
the  actual  removal  and  replacement  of  the  defective  part  would  be  all  that 
would  be  required.  Throughout  the  remainder  of  this  research,  a distinction 
will  be  drawn  between  external  and  internal  repair.  External  repair  will 
refer  to  repair  activities  associated  with  External  Case  components  and  in- 
ternal repair  will  designate  repair  activities  associated  with  parts  located 
within  the  "nested"  internal  assemblies. 


THE  KT-73  LOGISTICS  SYSTEM 

The  basic  components  of  the  maintenance  and  repair  system  for 
KT-73  IMU's  is  illustrated  in  Figure  2.  Block  1 represents  a KT-73 
that  is  performing  its  primary  function;  namely,  assistance  in  the  navi- 
gation of  A7-D  or  A7-E  aircraft. 

As  illustrated  Ly  Block  2,  built-in-test  equipment  is  used  to 
monitor  key  performance  measures  of  the  IMU  while  the  unit  is  still  in- 
stalled in  the  aircraft.  If  these  tests  indicate  the  unit  is  malfunctioning, 
the  IMU  is  removed  from  the  aircraft,  and  moved  to  the  field  maintenance 
activity  (Block  3)  for  further  testing. 

Field  maintenance  activities  consist  of  more  precise  testing  and 
diagnosis  than  can  be  performed  while  the  equipment  is  on  the  aircraft.  If 
these  tests  indicate  a unit  is  bad,  it  is  repaired  and  retested  in  the  field, 
if  possible  (Block  4).  When  the  field  tests  indicate  the  item  is  good,  it 
is  installed  in  the  aircraft  and  is  ready  for  pre-flight  checkout.  On  the 
other  hand,  if  field  repair  is  not  possible  or  appears  undesirable,  the  unit 
is  shipped  to  the  depot  (Block  5). 

At  present,  the  Aerospace  Guidance  and  Metrology  Center  (AGMC) 
is  the  only  depot  repair  location  for  KT-73  Inertial  Measurement  Units. 

As  illustrated  in  Figure  2,  the  major  aspects  of  depot  level  repair  at  AGMC 
Include : 
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a.  Receiving,  inspection,  and  diagnosis  of  incoming  units. 

b.  Repair  or  replacement  of  components  that  are  considered 
faulty. 

c.  Calibration  and  final  testing  of  the  overhauled  unit. 

d.  Shipment  to  the  field  installations. 

These  activities  are  illustrated  in  Blocks  6 through  12  of 
Figure  2. 

A more  detailed  flow  chart  and  associated  cost  data  for  KT-73 
depot  repair  activities  is  presented  in  Figure  3.  This  data  was  developed 
by  Watson  and  Waterman  (1974)  as  part  of  a cost  analysis  of  the  KT-73  re- 
pair process.  A majority  of  the  following  discussion  is  based  on  their 
work. 

In  examining  Figure  3 it  should  be  recognized  that  each  of  the 
16  major  "stages"  shown  actually  represents  one  or  more  activities.  The 
stage  shown  as  Repair  and  Replace  External  Module,  for  example,  actually 
covers  repair  or  replacement  of  four  different  modules,  as  well  as  a test 
to  ensure  that  the  fault  identified  has  actually  been  repaired.  Consequently, 
although  16  stages  are  shown,  the  work  performed  at  each  of  these  stages 
actually  covers  correction  of  a range  of  specific  malfunctions,  and  does 
not  represent  a set  of  "standard"  tasks. 

After  a KT-73  IMU  is  received  at  AGMC,  and  the  necessary  paper- 
work is  completed,  the  IMU  is  forwarded  to  the  KT-73  Repair  Shop.  All 
IMU's  arriving  at  the  shop  from  the  field  are  then  processed  through  the 
Receiving  Stage.  During  this  stage  a complete  check  of  the  IMU  is  per- 
formed using  an  automated  test  station,  to  confirm  the  malfunction  reported 
by  the  previous  user  and  to  identify  any  additional  problem  areas. 

Dependent  upon  the  results  of  the  receipt  tests,  the  IMU  is 
classified  as  requiring  either  internal  (cluster)  repair  or  repair  of  one 
of  the  external  modules.  If  internal  repair  is  required,  the  IMU  is  sent 
to  one  of  the  stages  at  the  next  level,  shown  on  Figure  3 as  Repair  and 
Replace  Internal  Module,  Repair  and  Replace  Torque  Motor  or  Electronics 
Repair  stages.  If  external  repair  is  required,  the  IMU  is  sent  to  Repair 
and  Replace  External  Module  Stage.  The  necessary  repair  is  then  performed 
and  the  IMU  is  tested  to  determine  whether  the  repair  was  successful. 
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FIGURE  3.  SIMPLIFIED  FLOWCHART  - KT-73  REPAIR  PROCESS 


Provided  that  the  results  of  the  test  show  that  the  original 
malfunction  has  been  corrected,  the  IMU  then  moves  to  the  next  stage  in  the 
repair  sequence.  The  actual  repair  sequence  for  an  IMU  is  dependent  upon 
the  faults  present  in  a particular  unit,  and  consequently  all  IMU's  do  not 
follow  the  same  path  through  the  process.  From  Figure  3 it  may  be  seen  that 
the  number  of  paths  which  a unit  may  follow  is  quite  large,  depending  on 
the  actual  faults  present.  This  progression  from  stage  to  stage  in  the  pro- 
cess continues  until  the  IMU  reaches  the  final  test  stages. 

Final  test  and  calibration  of  a repaired  IMU  is  carried  out  in 
the  IMU  Calibration  and  Final  Test  stages,  and  as  a result  of  the  tests 
conducted  during  these  stages  the  IMU  is  classified  either  as  "satisfactory" 
for  return  to  service,  or  as  "unsatisfactory"  and  requiring  further  repair. 
Each  of  these  stages  comprises  a number  of  specific  functional  tests.  The 
Final  Test  stage,  for  example,  includes  13  individual  tests,  each  designed 
to  evaluate  the  performance  of  the  IMU  in  a particular  phase  of  its  opera- 
tion. Successful  completion  of  all  13  tests  is  required  before  the  IMU 
is  classified  as  satisfactory  and  shipped  to  field  activities. 

Ideally,  items  would  progress  sequentially  from  one  stage  to 
the  next  until  the  IMU  reaches  the  final  test  stage  and  is  shipped  to  the 
field.  Unfortunately,  tests  at  a given  stage  may  uncover  problems  that 
were  introduced  earlier  in  the  repair  process;  such  faulty  items  must  be 
returned  to  the  earlier  stage  for  correction.  This  "feedback"  can  be  very 
expensive. 


Consider,  for  example,  the  case  where  an  IMU  which  has  been  repaired 
at  the  Internal  Module  Stage  reaches  the  Final  Test  Stage  and  is  found  to 
be  unsatisfactory.  If  the  unsatisfactory  condition  can  be  directly  diagnosed 
as  requiring  work  at  the  Internal  Module  Stage,  the  IMU  must  then  be  "fed 
back"  to  that  stage  for  correction  of  the  fault.  However,  because  of  the 
error  early  in  the  process,  the  unit  must  repeat  not  only  the  Internal  Module 
and  Final  Test  Stages,  but  all  intervening  stages  in  the  process. 

While  this  type  of  feedback  is  costly  enough,  next  consider  the 
same  situation,  but  assume  the  fault  is  improperly  diagnosed,  and  the  unit 
is  incorrectly  sent  to  the  Remove  and  Replace  Torque  Motor  Stage.  After 
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work  is  performed  at  this  stage,  the  unit  must  still  eventually  be  returned 
to  the  Internal  Module  Stage  to  correct  the  true  cause  of  the  fault. 

If  more  than  one  repair  or  diagnosis  error  occurs,  a number  of 
stages  may  have  to  be  repeated  more  than  once,  with  a corresponding  increase 
in  cost. 

The  above  discussion  illustrates  the  great  complexity  of  the  KT-73 
test  and  repair  system.  Let  us  now  review  the  current  state  of  the  art 
for  optimizing  decision  processes  in  such  complex  systems. 


II.  REVIEW  OF  THE  LITERATURE 


A critical  factor  in  the  design  of  any  maintenance  system  is  the 
precision  of  the  diagnosis  process.  If  an  automobile  has  a flat  tire,  a 
decision  to  change  that  tire  is  almost  certainly  correct.  However,  if  an 
inertial  measurement  unit  produces  large  navigation  errors,  the  actions 
needed  to  correct  the  condition  are  far  less  obvious.  For  complex  equip- 
ment, it  is  usually  very  difficult  to  determine  the  true  cause  of  a given 
operational  problem.  For  example,  large  navigational  errors  in  a KT-73 
Inertial  Measurement  Unit  (IMU)  may  be  caused  by  a faulty  A-200  accelera- 
meter;  there  are  several  of  these  in  a KT-73  unit.  Furthermore,  to  de- 
termine which,  if  any,  of  these  is  faulty  requires  complete  disassembly 
of  the  IMU,  a very  expensive  and  time  consuming  process.  To  make  matters 
worse,  it  is  possible  that  other  components  that  are  originally  good  may 
be  damaged  or  misaligned  during  the  disassembly  operation. 

Many  techniques  are  available  for  determining  cost-effective 
repair  decision  when  there  is  perfect  diagnosis,  but  much  less  is  known 
about  correct  maintenance  policies  in  the  face  of  diagnostic  uncertainty. 
Several  authors  have  modeled  the  latter  problem  as  a Markovian  decision 
process  with  probabilistic  observation  of  states.  This  includes  the  work 
of  Astrom  (1965),  Eckles  (1968),  Satia  and  Lave  (1973),  Smallwood  and 
Sondik  (1973),  and  White  (1976).  These  authors  explore  the  mathematical 
characteristics  of  optimal  decisions  for  these  processes,  and  propose 
computational  algorithms  for  determining  optimum  decisions;  however  the 
algorithms  appear  useful  for  only  very  simple  problems.  The  above 
authors  assume  that  all  transition  and  observation  probabilities  are 
known.  In  practice,  estimation  of  these  probabilities  can  be  a serious 
problem.  A non-linear  programming  approach  for  estimating  the  transition 
probabilities  for  unobserved  states  has  been  reported  by  Genet  (October,  1970). 
Unfortunately,  the  large  amount  of  computation  required  by  this  approach 
has  limited  its  usefulness. 

Another  approach  to  decision  making  with  imperfect  information 
is  straightforward  application  of  decision  and  utility  theory  techniques. 
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This  approach  uses  decision  trees  to  describe  the  structure  of  the 
decision  problem.  This  structuring  involves  identification  of  available 
alternatives,  their  possible  consequences,  and  the  conditions  under  which 
those  consequences  might  occur.  Utility  theory  is  then  used  to  develop 
scales  for  measuring  the  relative  desirability  of  actions  that  lead  to 
several  consequences  (e.g.  medical  "repair"  actions  may  lead  to  consequences 
such  as  "cure",  "no  effect",  or  "severe  complications").  Finally, 

Bayesian  inference  methods  are  used  to  combine  subjective  probability  esti- 
mates with  empirical  data  to  determine  a "best"  action.  Application  of 
these  techniques  to  medical  diagnosis  decisions  is  reported  by  Betaque 
and  Gorry  (1971).  Brown  and  McAllister  (1962)  and  Firstman  and  Gluss 
(1960)  discuss  use  of  these  techniques  for  fault  diagnosis  in  electronic 
equipment . 

Each  of  the  technical  approaches  described  above  should  prove 
useful  in  the  design  of  improved  diagnosis  and  repair  systems.  Markov 
Process  models  may  be  used  to  identify  some  (but  not  all)  of  the  key  de- 
cision points  in  a multi-stage  system  and  to  estimate  the  impact  of  pro- 
posed changes.  Such  a model  should  be  useful  in  determining  which  alter- 
natives should  be  studied  in  more  detail. 

The  decision  theory  approach  appears  to  have  excellent  potential 
for  improving  specific  diagnostic  and  repair  decision.  The  theory  is  well 
developed;  practical  application  appears  to  depend  on  development  of  data 
processing  tools  to  implement  the  calculations  and  on  development  of  de- 
cision trees  describing  the  technology  of  specific  repair  processes. 

In  theory,  decision  trees  could  be  developed  to  aid  in  the 
design  of  multi-stage  test/repair  systems.  In  practice,  however,  the  ex- 
treme complexity  of  most  Air  Force  equipment  will  probably  limit  usefulness 
of  this  technique  to  the  design  of  single  diagnosis  and  repair  stages. 

In  summary,  no  single  technique  (or  set  of  techniques)  currently 
exists  for  determining  optimum  diagnosis  and  repair  systems.  However,  tools 
for  attacking  pieces  of  the  total  problem  ao  exist.  In  the  following  sections, 
we  will  illustrate  the  use  of  these  tools  in  the  analysis  of  specific  diag- 
nosis/repair decisions. 
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III.  UNIVARIATE  TESTS 


Suppose  that  the  condition  of  a given  item  can  be  classified 
into  one  of  two  possible  states:  operational  or  failed.  If  the  item 

is  operational,  we  assume  it  is  capable  of  performing  assigned  missions 
within  the  item's  design  specifications;  if  failed,  the  item  cannot  per- 
form to  these  requirements.  For  simplicity,  we  will  say  the  item  is  "good" 
if  it  is  operational,  and  "bad"  if  it  is  not. 

Now  suppose  that  we  cannot  directly  observe  whether  the  item 
is  good  or  bad.  Rather,  we  can  only  estimate  the  item's  condition  by 
performing  a series  of  electrical  and  mechanical  measurements.  Suppose 
the  net  result  of  these  actions  is  summarized  in  a single  number,  which 
we  will  call  the  "reading".  As  noted  by  Genet  ("Avionics  Cost  Reduction 
Through  Improved  Tests") , variations  in  individual  readings  from  test  to 
test  are  often  due  to: 

a.  Variability  within  the  item  itself 

b.  Test  station  calibration  and  noise 

c.  Operator  variability. 

Variability  may  also  be  caused  by  interactions  among  these  sources 
or  by  other  causes. 

Figure  4 illustrates  a situation  in  which  there  is  no  varia- 
bility in  test  readings.  In  this  example,  a good  item  always  has  a reading 
of  100,  while  a bad  item  always  has  a reading  of  50.  This  is  an  example 
of  a perfect  test;  there  is  a one-to-one  correspondence  between  the  test 
reading  and  the  true  state  of  the  item. 

Figure  5 illustrates  a case  in  which  there  is  item  varia- 
bility but  no  other  source  of  variance  in  the  readings.  In  this  case, 
if  we  were  to  take  readings  on  a large  number  of  items  that  were  known  to 
be  good,  the  average  reading  would  still  be  100,  but  specific  readings 
would  vary  about  the  mean.  Similarly,  if  a large  number  of  bad  items  are 
measured,  the  average  value  is  50  in  this  case,  but  specific  items  would 
have  values  both  above  and  below  the  mean  due  to  differences  within  each 
item. 
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Now  let  us  consider  what  happens  when  there  are  still  more 
sources  of  reading  variability. 

Lee  i indicate  the  ith  reading  in  a series  of  readings,  and  let 

■ value  read  on  the  test  equipment 
VQ  ■ average  value  of  a good  item 

» average  value  of  a bad  item 
I1  * item  error,  or  "noise",  from  the  "true  value"  of  the 
item 

* station  error 
0^  ■ observer  error 

■ unexplained  error;  i.e.  error  due  to  other  than  item, 
station,  or  observer  error,  including  interaction 
effects  among  items,  stations,  and  observers. 

Then,  if  an  item  is  good. 


R1  ■ vo  + !1  + Si  + °i  + V1 


(X) 


The  expected  value  of  the  reading,  E(R),  is  then 

E(R±)  = VG  + E(S1)  + E(01)  + E(Ui)  (2) 

since  E(I^),  the  expected  value  of  item  error  for  a good  item, 
is  zero  by  definition  of  VG  + 1^.  Similarly,  if  an  item  is  bad , 

E(Rt)  - Vfi  + E(St)  + E(0i>  + E(U±)  (3) 

If  [E(Si)  + ECO^)  + EftJ^)]  * 0,  the  test  is  unbiased,  since 
the  expected  reading  then  equals  the  true  mean  value.  Otherwise,  the  test 
is  biased.  If  a test  is  biased,  an  unbiased  reading  may  be  obtained  by 
simply  subtracting  the  amount  of  bias  out  of  the  reading.  This  is  essentially 
what  happens  when  equipment  is  calibrated.  In  our  remaining  discussions,  we 
assume  that  specific  tests  are  unbiased. 

2 2 2 

Let  os  , oQ  , Oy  denote  respectively  the  variance  due  to  station, 

2 2 

observer,  and  unexplained  error  sources.  Also,  let  and  a0  denote  the 

2 

respective  items  variances  of  good  and  bad  units.  Since  oy  includes  all 
unexplained  variation,  including  interactions  effects,  for  good  items 


+ 


(4) 


while  for  bad  items 


2 2 
o_  + a„ 

B S 


2 ^ 2 
+ ”0  + °u 


(5) 


Hence,  if  we  know  the  average  values  and  variances  for  good  and 
2 2 

bad  items  (V^,  , and  Vg,  a g , respectively)  and  the  sum  of  variances 

2 2 

Og2  + oQ  + Oy  due  to  station,  operator,  and  unexplained  error  sources, 

we  can  compute  the  mean  and  variance  of  reading  errors  for  good  and  bad 
items.  Note  that  we  do  not  need  to  know  the  individual  components 


Og  , 0^  , or  Oy  ; only  the  sum  is  needed.  To  determine  the  variance  of 
the  sum  is  a much  easier  estimation  problem  than  estimating  the  characteris- 
tics of  each  variance  component  individually. 

If  each  of  the  error  sources  are  normally  distributed,  specific 
readings  will  also  be  normally  distributed  with  the  parameters  given  by 
(2)  - (5)  above.  Figure  6 illustrates  how  Figure  5 might  change  if 
additional  error  sources  are  present.  With  more  error  sources,  the 
variance  is  higher,  and  hence  the  distribution  of  readings  is  less  peaked; 
there  is  also  a wider  range  for  possible  observations. 

Note  that  the  distribution  of  readings  for  good  and  bad  items 
overlap  in  Figure  6.  For  example,  some  good  items  can  have  readings  of 
less  than  80,  while  some  bad  items  can  have  readings  of  more  than  80. 

Here  we  no  longer  have  a perfect  test.  There  is  no  longer  a one-to-one 
correspondence  between  a reading  and  a good  or  bad  item.  However,  the 
test  is  still  useful  for  distinguishing  between  good  and  bad  items,  since 
the  area  of  overlap  is  very  small. 

Figure  7 illustrates  a case  where  the  distribution  of  readings 
overlap  significantly.  In  this  case,  a specific  reading  tells  us  very 
little  about  the  true  state  of  the  item  being  tested.  If  the  two  distri- 
butions overlap  completely,  the  reading  tells  us  nothing  about  the  item's 
true  condition.  Such  a test  is  said  to  be  invalid. 
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CUTTING  SCORES  AND  OPERATING  CHARACTERISTIC  CURVES 


One  method  of  standardizing  test  decision  making  is  through  the 
use  of  cutting  scores.  For  example,  suppose  that  for  the  item  illustrated 
in  Figure  6,  a cutting  score  of  80  is  established.  In  testing  a given 
item,  a specific  reading  R will  be  obtained.  If  this  reading  is  greater 
than  the  cutting  score,  80,  the  item  passes  the  test;  and  is  then  treated 
as  if  it  were  a good  item.  If  the  reading  is  less  than  80,  the  item  would 
be  assumed  to  be  bad,  and  appropriate  repair  actions  would  then  be  taken. 
Note  from  Figure  6 that  with  a cutting  score  of  80,  some  bad  items 
will  pass  the  test,  and  some  good  ones  will  fail.  On  the  other  hand,  if 
the  cutting  score  is  set  at  100,  no  bad  items  will  pass  the  test,  but  many 
good  items  will  fail  it. 

One  way  to  quantify  the  sensitivity  of  a given  item  to  different 
cutting  scores  is  shown  in  Figure  8.  This  figure  plots  the  probability 
of  passing  the  test  versus  various  cutting  scores.  Two  curves  are  shown: 
one  for  items  that  are  in  fact  good,  another  for  bad  items.  We  shall  re- 
fer to  these  and  similar  curves  as  the  characteristic  curves,  or  C-curves . 
for  the  test,  since  the  curves  characterize  the  power  of  the  test  to  dis- 
criminate between  good  and  bad  items.  In  constructing  Figure  8,  we 
assumed  that  VQ  * 100;  a G » 10;  V0  « 50;  » 15.  This  is  the  same 

data  used  to  construct  Figure  -6. 

Note  that  with  a cutting  score  of  80,  there  is  a 2.3%  chance  of 
passing  a bad  item,  while  there  is  also  a 2.3%  chance  of  failing  a good 
item.  On  the  other  hand,  with  a cutting  score  of  100  no  bad  ones  will 
pass  the  test;  but  50%  of  all  good  items  will  fail  it. 
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FIGURE  6.  DISTRIBUTION  OF  READINGS  WITH  SERVERAL  ERRORS 
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ILLUSTRATION  OF  AN  INVALID  TEST 
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IV.  OPTIMUM  CUTTING  SCORES 


In  setting  cutting  scores,  two  types  of  costs  are  incurred. 

If  the  cutting  score  is  set  too  high,  some  good  items  will  fail  the 
test,  and  will  mistakenly  be  sent  to  undergo  repair  or  overhaul  actions. 

On  the  other  hand,  if  the  cutting  score  is  set  too  low,  some  bad  items 
may  pass  the  test,  and  may  thus  jeopardize  the  success  of  any  mission  in 
which  it  is  used.  Hence,  in  comtemplating  the  establishment  of  cutting 
scores  two  important  questions  arise.  The  first  is,  "what  is  the  cutting 
score  which  minimizes  the  cost  of  decision  errors?"  The  second  question 
is,  "How  can  we  decrease  the  variance  of  the  test  so  that  this  undesirable 
tradeoff  can  be  avoided  completely?"  The  second  question  requires  analysis 
of  the  technology  of  the  specific  test  involved  and  is  thus  beyond  the 
scope  of  this  paper.  Hence,  the  remainder  of  this  section  will  focus  on 
the  question  of  optimizing  cutting  scores  in  a given  test  environment. 


TEST  FACILITIES  WITH  KNOWN  PENALTY  COSTS 

Consider  a single  test  facility  such  as  that  illustrated  in 
Figure  9 that  tests  incoming  items.  Assume  these  items  are  either  good 
or  bad;  there  is  no  intermediate  level  of  deterioration.  As  noted  earlier, 
two  types  of  errors  can  be  made  here:  (a)  passing  a bad  item  or  (b)  fail- 

ing or  rejecting  an  item  that  is,  in  fact,  good.  Suppose  there  are  known 
economic  penalties  associated  with  each  of  these  errors;  specifically,  let 
CRG  ■ cost  of  rejecting  a good  item 
CAB  * cost  of  accepting  a bad  item. 

For  example,  if  a good  item  fails  a field  level  test,  it  may  be 
sent  to  the  depot  for  repair.  CRG  then  represents  the  cost  of  transporting 
the  item  to  and  from  the  depot  plus  the  cost  of  work  performed  at  the  depot 
level.  On  the  other  hand,  if  a bad  item  passes  the  test  the  success  of  sub- 
sequent missions  may  be  jeopardized.  CAB  should  then  represent  an  economic 
measure  of  the  cost  of  a jeopardized  mission.  [Another  approach  is  to  treat 
CAB  as  a parameter,  and  then  develop  cost /effectiveness  curves  that  relate 
probability  of  mission  success  to  logistics  support  costs.]  If  passing  a 
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bad  item  involves  risks  to  human  life  or  other  significant  consequences, 
utility  theory  techniques  may  be  used  to  develop  estimates  of  CAB.  For 
an  example  of  this  approach,  see  Betaque  and  Gorry  (1971). 

For  the  remainder  of  this  section,  we  assume  that  numerical 
values  of  CRG  and  CAB  are  available.  We  wish  to  determine  the  specific 
cutting  score  X that  minimizes  the  total  expected  cost  of  decision  errors. 
The  probability  that  an  incoming  item  is  an  fact  good,  P(G),  is  also 
assumed  known. 

Suppose  we  are  considering  setting  the  cutting  score  at  a 
value  of  X.  The  total  expected  cost,  EC(X),  of  using  a cutting  score  of 
X is  then 


EC (X)  = E[Cost  of  accepting  a bad  unit]  + E[Cost  of 
rejecting  a good  unit]. 


(6) 


Hence, 


EC (X)  = CAB  • P[(R>X)  and  (unit  is  bad)]  + 

P[(R<X)  and  (unit  is  good)]. 


CRG 


P (R| B)P (B)dR  + / CRG-P  (R|G)P(G)dR  (7) 
R<X 


= / CAB 
R>X 

* CAB-P (B)  / P(R|B)dR  + CRG-P  (G)  • / P(R|G)dR  (8) 
R>X  R<X 


The  minimum  cost  cutting  score  occurs  when  the  derivative  of  EC(X) 
with  respect  to  X is  zero.  Hence, 

dEC (X)  - 0 - -CAB-P(B)P(X|B)  + CRG-P(X|G)  = 0.  (9) 

dX 


Hence,  for  the  optimum  cutting  score, 

pfvlRN  CRG-P (G)  P(X|G) 
FtX|B>  - CAB.p(B) 

which  may  be  written 

P (X  | B)  - K-P(XjG)  where  K = 


(10) 


(ID 


H HI  mat 
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Equation  (11)  simply  says  that  the  optimum  cutting  score  X is  such  that  the 
conditional  probability  of  a reading  X given  the  item  is  bad  just  equals 
a constant  (K)  times  the  conditional  probability  of  a reading  X given  the 
item  is  good.  The  constant  K is  the  ratio  of  the  cost  of  rejecting  all 
good  items  [CRG*P(G)]  to  the  cost  of  accepting  all  bad  items  [CAB*P(B)j. 

Equation  (11)  provides  a necessary  condition  for  an  optimal  cut- 
ting score;  note  that  this  result  is  independent  of  the  form  of  the  proba- 
bility distribution  involved.  To  determine  sufficient  conditions  for  an 
optimal  cutting  score,  however,  the  second  derivatives  of  the  cost  function 
EC(X)  must  be  investigated.  In  the  next  section,  we  present  both  necessary 
and  sufficient  conditions  for  the  case  of  normally  distributed  error  sources. 


Normally  Distributed  Test  Errors 


Let  us  assume  that  the  conditional  distributions  of  P(R|B)  and 

P(R|G)  are  normally  distributed  with  means  and  variances  of  W , a ^ and 

B B 


WG,  aG  for  bad  and  good  units,  respectively;  that  is,  P(R|b) 

X—^B  ^ W 2 

exp  [-1/2  (-5—)  ] and  P(R|G)  - SzVa  exp  [-1/2  (^=-£) 


Equation  (11)  may  then  be  written  as 
1 X-WR  2 


2^B  exP  t1/2  £—>  1 - K exfctl/2  (^M 

B JZttG  aG 


which  may  be  rearranged  into  the  form: 

y 2 0o  W 2 

exp  [-1/2  (— — ) ] - K*  — exp  [-1/2  (~£)  ] 
B G °G 


Talcing  natural  logarithms. 


X-WB  2 
-1/2  (- — — ) 

aB 


Hence, 


k»b  x-wg  2 

lo  — - 1/2  'V’ 


X2-2«b.X  + V 


-2  ln  ^ + 


X -2WgX  + oc2 


X -2X(— 
0G 


— ) + (— 
2 ’ K 2 

°B  0G 


-21n 


(**) 
v aG; 


0 


(16) 


Let 


A 


B 


C 


2 In  (K  ~) 


(17) 


(18) 


(19) 


Using  (17)-(19) , equation  (16)  may  be  written  as: 
AX2  + BX  + C = 0 


(20) 


From  the  quadratic  formula,  values  of  X which  satisfy  (20)  are  given  by 


X 


-B  + 


/b  -4AC 
2A 


(21) 


assuming  A ^ 0. 

Equation  (21)  identifies  values  of  X for  which  the  slope  of  the 
expected  cost  curve  is  zero.  To  determine  if  the  associated  cost  is  a local 
maximum  or  minimum,  we  may  examine  the  second  derivative.  Specifically, 
since 


then 


dEC  (X) 
d 


AX  + BX+C 


d2EC(X) 

d 


* 2AX  + B 
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(22) 


(23) 


From  calculus,  if  X*  satisfies  (21),  X*  is  a local  minimum  if 
(23)  is  negative.  If  (23)  is  positive,  X*  is  a local  maximum:  finally,  if 
(23)  is  zero,  X*  is  an  inflection  point,  and  is  neither  a local  maximum 
nor  a local  minimum. 

Example  1 


To  illustrate  the  use  of  the  above  formulas,  consider  the 
situation  illustrated  to  Figure  10.  In  this  example,  the  failure 
variance  is  larger  than  the  variance  for  good  units,  and  99%  of  the  units 
being  inspected  are  good.  This  would  be  the  case,  for  example,  for  "on- 
aircraft"  tests  of  items  with  low  failure  rates.  For  this  example,  assume 
the  cost  of  accepting  a bad  unit  (CAB)  is  $500,  while  the  cost  of  rejecting 
a good  unit  (CRG)  is  $200. 

Using  the  data  from  Figure  10 


A - 

(^2  * 

. (-1 L-) 

v 2 j' 

3 

0075 

(24) 

o„ 

a 

10  20 

G 

B 

W 

W 

B - 

-2  (- ^ 

- -5-) 
2 1 

- -2  m - 

10_) 

2' 

= -1.75 

(25) 

°G 

°B 

10 

2GT 

W 2 
WG 

WB2 

Ka_ 

1002 

502 

C - 

2 

°G 

2 In  ( ) - 

°G 

102 

202  " 

2 In 

20 

-21n  (39.6  -=g)  - 85.01  (26) 


since 


_ CRG*P (G)  200  .99  , 

CAB-P(B)  500  .01  J 


(27) 


Therefore,  equation  (16)  yields 


-B  +^2-AAC 
2A 

68.9;  164.4 


1.75  + /(1.75)2  - 4(.0Q75)  (88.67) 
2 (.0075) 
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(28) 


The  cvo  candidates  for  the  minimum  cost  cutting  score  are  thus  X*68.9  and 
X-164.4.  When  X - 68.9,  2AX+B  - 2 • (. 0075) • (68. 9)  - 1.75  - 
-.716,  a negative  value. 

Hence,  X*68.9  is  a local  minimum.  When  X*164.4,  2AX-B*+.717,  a positive 
value;  hence,  X«164.4  is  a local  maximum. 

In  this  example,  the  expected  cost  EC(X)  is: 

EC (X)  - CAB«P(B)  / P(R|B)dR  + CRG*P(G)  / P(R|G)dR  (29) 

R>X  R<X 

-5 • / P(R|B)dR  + 198*  f P(R|G)dR  (30) 

R>X  R<X 

The  expected  cost  for  this  example  is  plotted  in  Figure  11  and  tabulated 
in  Table  1.  As  can  be  seen  from  the  figure,  the  minimum  cost  occurs  at 
X-68.9;  however,  the  expected  cost  is  very  close  to  the  minimum  for  cutting 
scores  in  the  range  60<X<75. 

Example  2 


Let  us  now  suppose  that  95%  of  the  items  coming  to  the  test 
station  are  in  fact  bad.  This  would  be  the  case  faced  by  a field  repair 
organization  when  fairly  reliable  Built  in  Test  Equipment  (BITE)  is  used 
to  determine  if  a "squawked"  item  is  to  be  removed  from  the  aircraft  and 
sent  to  field  repair.  Using  the  data  from  example  1 for  the  other 

EC(X)  = CAB*P(B)*/  P(R|B)dR  + CRG*P(G)*/  P(R|G)dR  (31) 

R>X  R<X 

= 500* (.95) •/  P(R|B)dR  + 200-(.05 )•/  P(R|G)dR  (32) 

R>X  R<X 

- 475  f ?(R| B)dR  + 10/  P(RlG)dR  (33) 

R>X  R<X 

Specific  values  of  EC(X)  are  tabulated  in  Table  D-2,  and  plotted 
in  Figure  12.  By  inspection  of  Figure  12,  the  minimum  expected  cost 
occurs  with  a cutting  score  of  about  X ■ 100.  Let  us  see  if  this  agrees 
with  the  analytical  solution. 
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FIGURE  11.  EXPECTED  COST  PER  UNIT  TESTED  FOR  EXAMPLE  1 
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FIGURE  12.  EXPECTED  COST  PER  UNIT  TESTED  FOR  EXAMPLE  2 


Evaluating  the  parameters  A,  B,  and  C,  we  find 


A - .0075 

B - -1.75 


(as  in  example  1) 


v CRG-P(G)  200. (.05) 
K * CAB-P(B)  = 500-(.95) 


Hence , 


Finally, 


. W?2..  - 2 ln((K  ' - 100  - 6.25  - 2 In  (.042)  = 100.09 


„ 2 2 
°G  °B 


+1.75  ±/ (1.75)  - 4 *(.0075)  (100.09) 

2 • ( .0075) 


1.75  ± .2445 
.015 


Hence, 


X - 100.4;  133.0 


Checking  the  second  derivatives;  2AX  + B * .244  when  X * 100.4; 

hence,  X ■ 100.4  is  a local  minimum.  When  X * 133.0,  2AX  + B * +.245; 

thus  X * 133.0  is  a local  maximum. 


Special  Case:  0 7 

or 


are  equal,  l.e.,  a 


2 2 

Consider  the  special  case  in  which  the  item  variances  o and  a 
_ 2 2 2 G B 


— •*  Vfc 


►» 


In  this  case,  coefficient  (12)  becomes  zero.  Hence,  (15)  may  be  written 
as 

BX  + C - 0 
which  gives 

X - -C/B 


W W„  2 

G+  B g iq  k 

2 WB  - 


Hence,  when  the  item  error  variances  are  equal,  the  optimum  cutting  score 
is  first  set  midway  between  the  two  means  (the  first  term  on  the  r.h.s)  and 
then  shifted  in  a direction  determined  by  the  factor  K.  If  the  cost  of  re- 
jecting good  items  is  greater  than  the  cost  of  accepting  bad  items.  In  K will 
be  positive,  and  the  optimum  x will  be  less  than  the  midpoint.  Conversely, 
if  the  cost  of  accepting  bad  units  is  greater.  In  K will  be  negative,  and 
the  optimum  X is  set  above  the  midpoint. 

For  example,  suppose 

WG  = 100  CRG  = 200  P (G)  - .99 

WR  =50  CAB  * 500  P (B)  = .01 

o = 10 


Then 


„ . ,CRG*P(G) . . 200 

K ln  (CAB^P(B)  = ln 


.99 


500  *.01 


ln  (39.6)  = 3.68 


The  optimum  cutting  score  X is  then 


100+50 


2- (10) 


100-50 


(3.68) 


75-15=60 


Hence,  for  this  situation,  the  optimum  cutting  score  is  60,  a value 
15  points  below  the  midpoint  of  the  mean  readings  for  good  and  bad  items. 
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Parallel  Test  Facilities 

The  preceding  analysis  considered  only  single  test  facilities. 

In  many  Instances,  however,  several  test  stations  will  work  in  parallel. 

In  this  section,  we  ask  "What  is  the  relationship  between 
cutting  scores  and  the  probability  of  passing  the  test  for  parallel  test 
facilities?"  As  we  shall  see  below,  if  we  know  the  discriminating  power 
of  individual  stations  (as  quantified  by  its  C-Curve,  as  in  Figure  8), 
then  we  can  compute  characteristic  curves  for  the  parallel  facilities  taken 
as  a whole. 

This  will  allow  us,  for  analysis  purposes,  to  treat  a set  of  parallel 
stations  as  if  it  were  a single  facility.  This  means  that  the  formulas  de- 
veloped in  the  previous  section  may  be  used  to  compute  optimum  cutting 
scores  for  both  single  and  parallel  facilities.  In  the  remainder  of  this 
section,  we  3how  how  to  compute  the  C-Curve  for  parallel  facilities  when  the 
C-Curve  for  the  individual  test  stations  are  known. 

Consider  a test  facility  consisting  of  two  parallel  test  stations, 
each  with  a known  C-Curve.  We  assume  that  the  probability  that  an  incoming 
item  is  assigned  to  a specific  station  is  independent  of  the  specific  con- 
dition of  the  item.  This  would  be  true,  for  example,  if  station  assignments 
are  based  largely  upon  which  station  has  the  least  amount  of  work  in  front 
of  it. 

Let  us  define  the  following  events. 

Event  Meaning 

item  is  tested  on  station  i 
G item  is  good 

B item  is  bad 

P item  passes  test 

F item  fails  test 

Also,  let  P(X)  denote  the  probability  of  X and  let  P (X  |E)  denote  the  con- 
ditional probability  of  X given  event  E has  occurred.  Finally,  let  P(X&E) 
denote  the  joint  probability  of  events  X and  E occurring  together.  First, 
let  us  compute  the  probability  that  a bad  item  passes  the  test.  Let 
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P [Bad  item  passes  test]  * P [ P j B ] 


(34) 


But  from  elementary  probability  theory, 

P[P|B]  - P[B  & P] 

P (B) 

and 

P[B&P]  - P[P&B|S1]P(S1)  + P[P&B|S2]P(S2)  (36) 

The  conditional  probabilities  on  the  right  may  bt  computed  since  the  C- 
Curves  for  each  station  are  known,  giving 

P[P&B|S.]  = PfP&B&S^]  - P[P|B&S^ ]P[B&S^ ] = P[P|B&S^ ]P(B)  (37) 

P(si)  P(St) 

since  P(B&S^)  - P(B)*P(Si)  assuming  random  station  assignment. 

Re-writing  (37),  we  have 

P[P&B|Si]  = P[P | B&S±]P (B)  (38) 

Finally,  substituting  (38)  into  (36),  and  the  result  into  (35),  we  have 

P[P | B]  = P[P|B&Si]P(B)  P(S1)  + P[P|B&S2]P(B)P(S2)  (39) 

P(B) 

- p[p|B&si]P(si)  + p[p|b&s2]p(s2) 

Equation  (39)  simply  states  that  the  probability  of  passing  a bad  item  for 
the  combination  of  stations  may  be  obtained  by  weighting  the  probability 
that  a given  station  passes  a bad  item  by  the  probability  that  that  station 
is  used,  and  summing  these  products  for  all  stations. 

Note  that  if  both  stations  have  identical  C-Curves,  P[P|b&S^]  - 

P[P|B4S2],  and  the  P[ passing  a bad  item]  for  the  combination  of  stations  is 

equal  to  the  probability  of  passing  a bad  item  for  an  individual  station. 

These  results  are  easily  generalized.  For  N parallel  stations, 
with  random  station  assignment. 

P[Passing| Bad]  - IN  P[P | B&SjP (S^  (40) 

i-1 

- IN  P[F|G&Si]P(Si) 


Similarly, 

P[ Failing | Good] 


i-1 


(41) 


V.  MULTIVARIATE  TESTS 


To  this  point  we  have  assumed  that  the  result  of  a given  test  was 
summarized  in  a single  reading  R.  Let  us  now  consider  more  complex  tests 
in  which  several  physical  or  electrical  measurements  of  a given  item  are 
obtained.  Based  on  these  measurements,  we  are  to  either  accept  or  reject 
the  item.  As  before,  we  assume  that  the  item  is  in  fact  either  good  or 
bad,  but  that  the  true  state  or  condition  of  the  item  cannot  be  directly 
observed.  The  question  is,  "How  can  optimum  cutting  scores  be  established 
in  this  case?" 

Let  r^  denote  the  value  of  the  ith  measurement  or  reading  for 
a particular  item,  and  let  R denote  the  vector  of  measurements  obtained 
during  a given  test.  That  is, 

R * (ri»  r2’  rip 

where  N denotes  the  number  of  measurements  taken.  Let  CAB  denote  the  cost 
of  accepting  a bad  item,  and  let  CRG  represent  the  cost  of  rejecting  a good 
item.  Then  the  following  decision  table  describes  the  current  situation. 


Item  is 
in  fact 

Good 

Bad 


Decision  Table  Given  a Reading  Vector  R 


Probability 
P(G |.R) 

p(b|r) 


Possible  Actions 


Accept 

Item 

0 

CAB 


Re j ect 
Item 

CRG 

0 


In  the  above  table,  P(G|R)  denotes  the  conditional  probability  that 
the  item  is  good  given  the  reading  vector  R is  observed,  while  P(B|R)  denotes 
the  conditional  probability  the  item  is  bad. 

Using  the  Bayes  decision  criteria,  we  would  accept  the  item  if  the 
expected  loss  due  to  accepting  bad  items  is  less  than  the  expected  loss  due 
to  rejecting  good  items.  That  is. 


accept  if 

CAB* P (B | R)  < CRG*P(G|R) 
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(42) 


BuC  the  probability  the  item  is  bad  equals  one  minus  the  probability  the 
item  is  good,  i.e. 

P(B|R)  = 1 - P(G|R)  (43) 

Therefore,  we  would  accept  if 

CAB* [1  - P (G | R) ] < CRG*P(G|R)  (44) 

Rearranging,  we  obtain 


> CAB  (45) 

CAB+CRG 

Hence,  we  should  accept  the  item  if  the  conditional  probability  the  item  is 
good  exceeds  the  ratio  of  CAB  to  [CAB  + CRG] . Conversely,  if  ?(G|R)  is  less 
than  this  ratio,  we  would  reject  the  item.  Finally,  if  P(G|.R)  equals  this 
ratio,  we  would  be  indifferent  between  accepting  or  rejecting  the  item,  since 
the  expected  cost  of  each  decision  is  the  same. 

The  conditional  probability  P(G|R)  for  a given  vector  of  readings 
R may  be  computed  using  Bayes  theorem.  Specifically, 


p (G l r\  , P(RlG)P(G) 

V 1 ' P(R|G)P(G)+P(R|B)P(B) 

where  P (F  | G)  and  P (K | B)  denote  the  probability  distributions  of  readings  from 

good  and  bad  items,  respectively,  and  P(G)  and  P(B)  denote  the  respective 
a priori  probabilities  that  an  item  is  in  fact  good  or  bad. 

Suppose  X is  a set  of  readings  such  that 


P(G|X) 


CAB 

CAB+CRG 


(47) 


For  notatlonal  convenience,  let 
Z = P(GjX). 

Then  from  Bayes  theorem, 

z , P(xlG)P(G) 

P(X|G)P(G)+P(X|B)P(B) 

Rearranging, 

Z*P(x|G)P(G)+Z*P(XlB)P(B)  - P(X|G)P(G) 
Z*P(X|B)P(B)  - (1-Z)*P(X|G)P(G) 

Hence , 

P(X|B)  - • 1^-  * P(x|G) 

But  recall  that  Z-P(X|G)  - CAB  | [ CAB+CRG ] . 

that 

(1-Z)  _ CRG 

Z CAB 

166 


(48) 

(49) 

(50) 

(51) 

(52) 

Hence,  it  can  be  shown 

(53) 


m maam 


(54) 


Finally,  substituting  (53)  into  (52), 

p(xIb)  a ^"Ty§j  * P(XIG) 

which  may  be  written  as 


p(x|b)  - k*p(x|g) 

(55) 

where 

CRG*P (G) 

CAB*P (B) 

(56) 

The  above  result  is  the  multivariant  equivalent  of  the  necessary 
condition  for  optimum  cutting  scores  given  by  equation  (11)  in  Section  IV. 

In  both  the  single  and  multivariate  case,  we  seek  values  of  X such 
that  the  conditional  probability  of  observing  X for  a bad  item  equals  a con- 
stant K times  the  conditional  probability  of  observing  X if  the  item  is 
in  fact  good.  The  constant  K equals  the  ratio  of  the  cost  of  rejecting  all 
good  items  [CRG*P(G)]  to  the  cost  of  accepting  all  bad  ones  [CAB*P(B)]. 

In  the  single  measurement  case,  relation  (55)  will  yield  a unique 
value  for  X for  most  probability  distributions  encountered  in  practice. 

In  the  multivariate  case,  however,  we  may  expect  a large  number  of  sets  of 
readings  X * (X^,  X^,  ...»  X^)  to  satisfy  the  above  condition. 

For  example,  consider  a test  in  which  two  readings,  r^  and  r^,  are 
taken.  Furthermore,  suppose  the  readings  are  statistically  independent.  Then 

P(R|B)  - P(r1|B)-p(r2|B)  (57) 

and 

P(R|G)  - P(r1|G)*P(r2|G)  (58) 

Let  X^  and  X2  denote  the  cutting  scores  to  be  used  for  comparison  with 
readings  ^ and  r2>  respectively.  Then  from  (55),  Xx  and  X2  must  satisfy 

P(X1|B)  P(X2|B)  - K-P(X1|G)-P(X2|G)  (59) 

If  a specific  value  for  X1  is  selected,  then  X2  must  satisfy 

P(X  |B)  - fPgj!g  * P(xJg)  (60) 

P(X1|B) 


Thus,  even  in  this  simple  case,  the 
for  the  second  reading  is  dependent 
first  reading. 


choice  of  the  optimum  cutting  score  X2 
upon  the  cutting  score  used  for  the 
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In  general,  specification  of  optimum  cutting  scores  for  a multi- 
variate test  requires  identification  of  all  sets  of  values  X ■ (X^,  X2>  .... 

X^)  satisfying  equation  (55).  In  the  two-reading  case,  this  may  be  done 
graphically,  as  illustrated  in  Figure  13.  Any  item  with  a combination  of 
readings,  lying  inside  the  "Acceptable  Region"  would  pass  the  test,  and  items 
with  readings  outside  this  region  would  fail.  In  this  example,  "rough"  cut- 
ting scores  may  be  established  using  the  extreme  measurements  in  the  Acceptable 
Region. 


FIGURE  13.  THE  ACCEPTABLE  REGION  FOR  A TWO-READING  TES' 


For  example,  in  Figure  13  if  the  measurement  Y is  taken  first, 

the  values  marked  Y^qw  and  ^ might  be  used  to  determine  if  measurement 

Z should  be  taken.  If  the  measurement  Y is  less  than  Y,  , more  than  Y,  . , , 

low’  high’ 

the  item  will  fail  the  test,  regardless  of  the  value  of  Z obtained.  Hence, 
if  Y is  outside  the  range  [Y^qw,  Y^  the  item  should  be  rejected  immediately 
On  the  other  hand,  if  the  reading  is  within  this  range,  Z must  also  be  measured 
to  determine  if  the  pair  of  readings  (Y,Z)  is  in  the  acceptable  region. 

When  more  than  two  readings  are  involved,  graphical  methods  can 
no  longer  be  used  to  specify  the  acceptable  region.  In  this  case,  two 
major  options  are  available. 

1.  Explicitly  specify  all  reading  vectors  X in  the  acceptable 
range,  using  tables  or  monographs 

2.  Determine  whether  to  accept  or  reject  an  item  by  first 
obtaining  the  reading  vector  R,  and  then  computing  P(G|R) 
directly. 


To  illustrate  the  second  approach,  suppose  CAB  ■ $500  and  CRG  3 $200. 
From  equation  (45),  after  obtaining  a given  reading  vector  R,  we  should 
accept  the  item  if 


CAB  _ 500 

P(GIR)  = CAB+CRG  " 5004-200 


.74 


(61) 


Hence,  if  the  conditional  probability  P(G|R)  > .74,  we  should  accept  the  item; 

otherwise,  we  should  reject  it. 

To  illustrate,  the  computation  of  P(G|R),  suppose  we  take  two 
measurements  r^  and  r^  that  are  bivariate  normally  distributed.  Specifi- 
cally, suppose  that  if  an  item  is  in  fact  good. 


P(R|G)  - K_  • exp  [-1/2 (R-U  ) * Sp  (R-U_) ] 


(62) 

(63) 


where  R denotes  the  reading  vector,  U denotes  the  vector  of  expected  readings 

G 

for  good  items,  V_  denotes  the  variance  - covariance  matrix  describing  the 
G -1 

joint  variability  of  readings  r,  and  r„,  and  S =V  by  definition.  (For  a 

1 L G G 

detailed  discussion  of  the  properties  of  multivariate  normal  distributions, 
see  F.A.  Graybill,  An  Introduction  to  Linear  Statistical  Models,  Volume  I, 
McGraw-Hill,  1961).  The  value  of  the  integration  K is  given  by 

TT  N X 1/2 

kg  * (64) 


where  X is  the  ith  characteristic  root  of  S„,  and  N denotes  the  number  of 
i G 

readings.  In  our  case,  N ■ 2. 

For  this  example. 


SG  " VG 


-1 


c 


01067 

00267 


.00267 

.01067 


(65) 


By  definition,  the  characteristic  root  X of  S„  satisfies,  for  any  vector  X, 

G 


which  implies 

lSG 


X X 


- I - 0 


(66) 


(67) 
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Hence,  for  SG> 


therefore, 
which  gives 
Hence , 


.01067- A 
.00267 


(.01067-  X )2  - (.00267) 2 


•01067-A  = + .00267 


.00800  or  .01334 


.00267 
.01067- a 


. 2 >1/2 
1-1  Xi 

2 it 


(.08944)  (.115499) 
2tt 


.0016441 


(68) 

(69) 

(70) 

(71) 

(72) 


Hence,  given  any  set  of  readings  r^,  and  r^,  we  may  evaluate  P(R]G)  by 
using  the  values  defined  by  equations  (63),  (65),  and  (72)  in  equation  (63). 

Similarly,  if  the  item  is  in  fact  bad,  let  us  assume  that  the  reading  vector 
R is  also  bivariate  normally  distributed,  with  the  following  parameters: 


where 


P(R|B)  - ^ exp  [-l/2(R-l£)'SB  (R-Ug ) ] 

-1 


°b  ‘(IS) ; 


/225  100\  S 

B " VJ.00  225/ ; 


V 

B - B 


.005538 

.00246 


In  this  case,  the  characteristic  roots  of  S satisfy 

O 


. 005538- X 
.00246 


.00246 
. 005538- X 


Hence , 


(.005538-  X )2  - (.00246) 2 


or 


.005538-  A 


+.00246 


Finally, 

Hence,  from  (64), 

s 


.00800  or  .00308 


(73) 

.00246"! 

.005538j 


(75) 

(76) 

(77) 

(78) 


(.00800)172  (.00308)172  (.089442)  f. 05548)  . 000790  (79) 

2n  2H 


Finally,  suppose  that  ve  obtain  readings  of  r^*85  and  r^'70,  and 
• .pp<  aa  that  th«  a priori  probability  that  the  item  is  good  is  P(G)  - .99. 


(74) 


1 •«*  »v».,at  lag  P(R  ,G)  using  equations  (62),  (63),  and  (72),  we  obtain 


R-U, 


85-100 

70-100 


(r-ug)’sg<r-ug) 


-15 
-30 

(-15,  -30) 
(-.24015,  -.48015) 


.01067 

.00267. 

"15l 

.00267 

.01067 

-30  J 

Therefore, 


-15 

-30 


18.006 


P(R|3)  * .0016441  exp  [-1/2(18.006)] 

= (.0016441)  (.000123) 

- .000  000  20222  =*  2.0222  x 10 


Also,  evaluating  P(r|b)  using  (73),  (74),  and  (79),  we  obtain 

85-50\  / 35 

75-50 j (25 


(R-U  )'S  (R-U  ) - (35,25)  1. 005538  /. 00246  1 

.00246  .005538 

• l J 


=•  (.255225,  .22455) 


=*  14.547 


Hence , 

P(R|B) 


Finally , 

P(G|R) 


(.000790)  exp  [-1/2  (14.547)] 
(.000790)  (.0006/4) 

.000  000  5482  - 5.482  x 10"7 


P(RlG)P(G) 

P(R|G)P(G)+P(R|B)P(B) 


(.000  000  2022)  (.99) 

(.000  000  2022)  (.99)  + (.000  000  5482)  (.01) 


2001.78 

2001.78  + 54.82 


.973 


Hence,  with  readings  of  85  and  70,  there  is  a 97%  chance  that  the  item  is  good. 
Note  that  our  a priori  probability  that  the  item  was  good  was  99%;  hence,  we 
have  revised  the  probability  that  this  particular  item  is  good  down  a little 
based  on  the  readings.  However,  97%  is  still  much  more  than  our  74%  acceptance 
limit;  hence,  this  item  would  be  accepted. 
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Now  suppose  a second  item  is  tested.  In  this  case,  suppose  the 
readings  are  r^  * 60  and  r ^ m 50.  In  this  case, 

R-U  * |60-100\  - / -40  I 

(50-100/  ( -50  j 

(R-U  )'S  (R-U  ) - (-40,-50)  (.01067  .00267  / -40  1 

(.00267  .01067  ( -50  j 

- (-.5603,  -.6403)  / -40  \ 

-50 


54.427 


Hence , 


P(R|r;)  - .001644  exp  [-1/2(54.427)] 

= (.001644)  (1.512  x 10-12) 


2.486  x 10 


Next,  let  us  compute  P(RjB).  We  obtain 


50-5d  - (101 

>0-50j  l Oi 


(R-U  )'S  (R-U  ) * (10,0)  .005538  .00246  10 

.00246  .005538  0 


(.05538,  .0246)  |10\ 


Hence , 


P(R|B)  - .000790  exp  [-1/2 (. 5538) ] 

- (.000790)  (.758131) 


Finally, 


P(G|R) 


.005989 


(2.48  x 10' 


(2.48  x 10-")  (.99)  + (.005989)  (.01) 

- 4.10  x 10'9  * 0 


Hence,  the  second  Item  is  almost  certainly  bad,  and  should  be  rejected. 


VI.  NETWORKS  OF  TEST  AND  REPAIR  FACILITIES 


The  previous  analysis  was  concerned  with  setting  optimum  cutting 
scores  for  a single  test.  For  many  complex  items,  however,  cost-effective 
logistics  support  requires  development  of  diagnostic  and  repair  facilities 
at  several  physical  locations.  In  this  section,  we  discuss  an  approach  for 
determining  optimal  cutting  scores  for  networks  of  test  and  repair  facilities. 


A FIELD-TO-DEPOT  FLOW  MODEL 


To  begin,  let  us  consider  an  airborne  item  supported  by  a logis- 
tics network  such  as  that  shown  in  Figure  14.  For  this  item,  tests  are 
performed  in  the  field  after  each  mission  is  flown.  If  no  problems  are 
discovered,  the  unit  remains  on  the  aircraft.  If  the  tests  indicate  a prob- 
lem, however,  the  unit  is  removed  from  the  aircraft  and  shipped  to  the 
depot  for  further  testing  and  repair.  After  all  known  problems  are  resolved, 
the  unit  is  returned  to  the  field  and  the  cycle  continues. 

Our  current  objective  is  to  estimate  the  cost  of  supporting  1,000 
sorties  if  a given  combination  of  cutting  scores  and  test  and  repair  facili- 
ties are  used.  Later,  we  will  discuss  how  to  determine  optimal  cutting  scores 
for  a network  of  facilities. 

To  simplify  our  discussion,  suppose  that  for  our  hypothetical 
item  the  same  tests  and  procedures  are  used  at  both  the  field  and  depot 
level.  Suppose  these  tests  have  the  following  characteristics: 

Value  Meaning 

50  Average  reading  for  failed  items 

100  Average  reading  for  good  items 

17  Standard  deviation  of  readings. 

Furthermore,  suppose  a cutting  score  of  75  is  used  at  both  field 
and  depot  levels.  Assuming  the  readings  are  normally  distributed, 

P [accepting  a bad  item]  * p|z  <.  — = .07 

P [rejecting  a good  item]  * P^Z  A — -07 
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Hence,  for  this  example,  a cutting  score  of  75  results  in  error 
probabilities  of  7%,  at  both  field  and  depot  levels,  while  the  chance  of 
a correct  decision  for  any  given  item  is  93%. 

To  continue  our  example,  suppose  that  the  repair  process  is  not 
perfect;  specifically,  let  us  assume  that  there  is  a 5%  chance  that  an  item 
that  is  sent  to  repair  will  be  bad  at  the  conclusion  of  repair  actions;  con- 
versely, there  is  a 95%  chance  that  repair  is  successful.  If  a good  item 
is  sent  to  repair  (due  to  testing  error) , we  assume  the  same  probabilities 
apply. 

Finally,  suppose  that  the  item  has  a mean  time  between  failures 

of  200  hours,  with  a constant  hazard  rate;  this  implies  that  the  probability 

the  item  is  used  for  at  least  X hours  before  failure  is  given  by 

- X 

P[no  failure  in  X hours]  ■ e 200 


If  the  average  sortie  length  is  two  hours,  this  implies  that  the  probability 
of  no  failure  on  any  specific  mission  is  e » .99;  hence,  the  pro- 

bability of  failure  is  .01. 

With  these  assumptions,  the  possible  conditions  of  the  unit  may 
be  described  by  the  state  diagram  shown  in  Figure  15.  In  this  diagram, 
circles  denote  particular  states  of  the  unit  while  arrows  denote  possible 
changes  of  state,  or  transitions,  that  the  unit  may  follow.  The  "G"  indicates 
states  in  which  the  component  is  in  fact  good,  while  "B"  indicates  states 
in  which  the  component  is  in  fact  bad.  Note  that  the  diagram  has  been  con- 
structed such  that  all  "good"  states  are  on  the  left,  and  all  "bad"  states 
are  on  the  right. 

The  following  symbols  denote  the  particular  physical  location  asso- 
ciated with  a given  item  state. 


Symbol 

S 

FT 

SD 

DT 

DR 

SF 


Physical  Location 

Installed  in  aircraft;  on  a mission 
Field  Test 

Ship  to  Depot;  item  is  in  transit 
Depot  test 
Depot  repair 
Ship  to  field 
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FLY  SORTIE 


FIELD 

TEST 


SHIP  TO 
DEPOT 


DEPOT 

TEST 


DEPOT 

REPAIR 

SHIP  TO 
FIELD 


FIGURE  .-15.  STATE  DIAGRAM  FOR  A SIMPLE  FIELD- TO-DEPOT  FLOW  MODEL 


i 


> 
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Hence,  Che  circle  marked  "S,G"  denotes  Che  state  in  which  the  item 
is  good,  and  is  flying  a mission.  Similarly,  the  state  "FT,G"  indicates 
the  item  is  being  tested  in  the  field  following  the  sortie,  and  is  still 
good. 

In  Figure  15,  the  numbers  beside  each  arrow  denote  the  pro- 
bability that  a given  path  is  followed  when  the  item  changes  state.  Hence, 
the  .01  by  the  arrow  connecting  states  "S,G"  and  "FT,B"  indicates  there  is 
a IX  chance  that  the  item  will  fail  on  any  given  sortie,  and  hence  bad  ("B") 
when  it  enters  the  post-sortie  field  test  ("FT").  Similarly,  the  .93  beside 
the  arrow  connecting  state  ("FT,G")  to  state  "S,G"  indicates  there  is  a 93% 
chance  that  a good  item  will  pass  the  field  test  ("FT");  while,  conversely, 
there  is  a 7%  chance  that  a good  item  will  mistakenly  fail  the  test  and 
be  shipped  to  the  depot  (state  "SD,G"). 


Evaluating  Expected  Costs 


This  flow  model  may  be  used  to  determine  the  expected  cost  of 
maintaining  a given  number  of  operational  units  with  a given  combination 
of  field  and  depot  cutting  scores.  Jointly  optimal  cutting  scores  may  then 
be  determined  by  evaluating  all  possible  combinations  of  cutting  scores  for 
the  field  and  depot  tests.  Optimization  procedures  might  also  be  used  to 
intelligently  select  specific  cutting  score  values  for  evaluation.  We  will 
discuss  one  such  optimization  approach  later  in  this  section. 

Let  us  now  consider  how  the  expected  cost  of  a specific  con- 
figuration of  cutting  scores  may  be  determined.  Suppose  we  assign  numbers 

1,  2,  N to  each  state  represented  by  a circle  in  Figure  15. 

Suppose  that: 


cost  of  passing  through  state  J 

expected  number  of  times  a unit  passes  through  state  j 

probability  that  a component  that  is  in  state  1 transi- 
tions to  state  j . 
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Then,  in  steady  state, 
N 

*1  = =1-1 


n p 

i ij 


for  j **  1,  2, 


N 


(80) 


That  is,  when  the  system  is  in  steady  state,  the  expected  number  of  times 
that  a unit  passes  through  state  j (i.e.  n^)  equals  the  expected  number 

of  times  the  unit  passes  through  state  i,  ( IK)  multiplied  by  the 
the  probability,  (P^)  that  a unit  next  transitions  from  state  i to  state 


nt  P^  are  then  summed  over  all  possible  prior  states. 


j . The  products  of 

Suppose  we  number  the  states  shown  in  Figure  15,  beginning  with 
state  "S,G" , and  continuing  until  all  states  are  identified.  Hence,  It^  de- 
notes the  number  of  successful  missions  flown.  If  the  system  of  equations 
(80)  is  solved  with  the  additional  requirement  that 


n. 


1000 


(81) 


IIj  denote  the  expected  number  of  times  that  a 


the  resulting  values  of 
given  unit  will  pass  through  state  j before  1000  successful  missions 
are  completed. 

For  example,  the  solution  to  the  system  of  equations  (80)  and  (81) 
using  the  data  from  Figure  15  is  shown  in  Figure  16.  The  numbers 
written  beside  each  state  j denotes  the  corresponding  value  of  11^  . For 
example,  notice  that  for  this  particular  configuration  of  tests  and  repair 
facilities,  for  every  1000  successful  missions  flown  there  will  be  69  good 
units  that  are  mistakenly  shipped  to  the  depot  for  repair,  and  an  average 
of  10.8  bad  units  shipped  to  the  depot.  Since  the  depot  test  is  also  im- 
perfect, we  would  expect  6 units  that  are  in  fact  good  to  be  mistakenly  sent 
to  repair.  In  contrast,  the  expected  number  of  bad  units  sent  to  repair 
is  10.8  per  1000  successful  missions.  Hence,  for  this  system  of  tests, 

6 (6  + 10.8),  or  362  of  all  repair  activity  is  on  good  items. 

If  the  cost  Cj  of  having  an  item  pass  through  each  state  j is 
known,  the  cost  of  supporting  1000  successful  missions,  C,  is  simply 

C * J Cj  nj  (82) 

For  example,  suppose  the  following  cost  calculations  apply  to  Figure  16. 
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Frequency  Cost  x 


Action 

Cost 

Per  1000  Missions 

Frequency 

Field  Test 

50 

1001.6 

50,080 

Ship  to  Depot 

200 

79.8 

15,960 

Depot  Test 

50 

96.6 

4,830 

Depot  Repair 

1,000 

16.8 

16,800 

Ship  to  Field 

200 

79.8 

15,960 

Penalty  for  flying 
with  a bad  unit 

20,000 

1.6 

32,000 

TOTAL 

135,630 

Hence,  for  this 

particular  test  and  repair  configuration,  the 

expected  cost  to  support  1000  successful  missions  is  $135,630,  or  $135.63 
per  mission. 


Optimizing  Networks  of  Tests 

The  above  results  apply  for  a particular  combination  of  cutting 
scores;  namely,  cutting  scores  of  75  at  both  the  depot  and  field  levels. 

Other  combinations  of  cutting  scores  may  produce  lower  overall  costs.  One 
approach  to  determine  the  best  combination  of  cutting  scores  is  to  simply 
repeat  the  above  analysis  procedure  for  all  interesting  combinations  of  field 
and  depot  cutting  scores.  Although  this  approach  is  not  very  elegant,  it 
can  be  easily  automated. 

Another  approach  to  determine  optimum  cutting  scores  is  to  use 
a variation  of  the  Markov  Decision  Process  Model  to  describe  our  decision 
problem.  (Wagner,  1969).  Figure  17  illustrates  how  the  state  diagram 
shown  in  Figure  16  might  be  transformed  into  a Markov  Decision  Model 
framework.  In  Figure  17,  five  possible  values  for  the  field  test  cutting 
score  are  identified.  These  are  denoted  by  the  letters  A,  B,  C,  D,  and  E. 
Similarly,  five  possible  depot  cutting  scores  (I,  J,  K,  L,  and  M)  are  identi- 
fied. 

Suppose  we  number  each  state  shown  in  Figure  17,  starting  with 
state  "S,G".  Let  C ^ , • 11^ , and  be  defined  as  above.  Then  the  optimum 
set  of  cutting  scores  may  be  determined  by  solving  the  following  mixed  inte- 
ger programming  problem  for  the  values  of  llj  and  Yi  that  minimize  C,  that  is 
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STATE  DIAGRAM  FOR  MARKOV  DECISION  PROCESS  MODEL 
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subject  to 


all  j 


given  to  the  constraints 


- U-Y 


i “ 


A,  B,  C,  D,  E,  (86) 
I,  J,  K,  L,  M 


+ + Y, 


Yi  + yj  + yk  + yl  + ym  = 1 (88) 

where  U denotes  a very  large  number,  say  1,000,000, 

1 if  cutting  score  i is  to  be  used 
Yi  = 0 otherwise  (89) 

G B 

and  and  X^  denote  the  values  of  I^,  corresponding  to  testing  a good  item 
and  b^d  item,  respectively,  using  cutting  score  i.  Equations  (87)  and  (88)  state 
that  one  cutting  score  must  be  selected  for  the  field  and  depot  levels,  respec- 
tively, while  equation  (86)  insures  that  if  cutting  score  i is  not  used,  the 
corresponding  IL  values  must  be  zero. 

To  solve  (83)-(89),  standard  mathematical  programming  systems  may 
be  used.  One  such  system  is  the  Mathematical  Programming  System-X  (MPS-X) 
developed  by  the  IBM  Corporation. 

The  above  procedures  may  be  applied  to  test/repair  systems  that 
are  much  more  complex  than  those  considered  so  far.  For  example,  Figure  3-8 
illustrates  the  state  diagram  for  an  item  which  may  be  tested  at  four  levels 
— one  test  in  the  aircraft,  one  test  in  the  field,  and  two  tests  at  the  depot. 
Figure  18  also  illustrates  the  item  can  be  repaired  at  two  levels — in  the 
field  or  at  the  depot. 

The  same  analysis  and  optimization  procedures  described  above 
may  be  applied  to  this  complex  system.  Even  more  complex  systems  may  be 
modeled  and  optimized  using  the  same  basic  methods. 


STATE  DIAGRAM  FOR  MULTI-LEVEL  TEST  AND  REPAIR 
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Data  Collection 


As  we  have  seen  in  the  preceding  sections,  optimization  of  main- 
tenance decisions  in  complex  repair  networks  requires  knowledge  of 

a.  The  variability  of  test  readings  as  a function  of  the  item's 
true  state.  Ideally,  this  should  be  quantified  by  the  C- 
Curve  for  each  test. 

b.  Failure  rate  characteristics  of  the  item. 

c.  The  reliability  of  each  repair  process;  i.e.,  the 
probability  that  repair  actions  will  produce  a "good" 
item. 

d.  Structure  of  the  test  and  repair  network,  and  the  costs 
per  unit  of  all  major  activities  in  this  network. 

In  most  cases,  much  of  the  above  data  will  not  be  available  from 
existing  data  sources.  The  main  body  of  this  report  describes  experimental 
design  and  cost  analysis  procedures  for  obtaining  such  information.  However, 
applying  these  techniques  can  be  expensive;  what  is  needed  is  a tool  for 
identifying  areas  in  which  there  is  a high  potential  payback  for  a detailed 
study  effort.  The  Markov  Modeling  procedure  may  be  slightly  modified  to  pro- 
vide such  a tool.  This  is  the  subject  of  the  next  section. 


VII.  PROCESS  FLOW  MODELS 


Repair  systems  for  the  KT-73  and  for  many  other  Air  Force  items 
are  extremely  complex;  involving  many  specific  repair  activities  and  diag- 
nostic and  performance  tests.  A fundamental  problem  in  improving  the 
effectiveness  of  these  complex  systems  is  identification  of  areas  with  high 
payback  potential,  that  is 

"...[to  determine]  the  general  areas  of  subprocesses  which 
would,  if  improved  slightly,  result  in  major  improvements 
in  cost  effectiveness,  and  conversely,  [to  determine]  areas 
or  subprocesses  which,  if  greatly  improved,  would  have  little 
effect  on  overall  cost  effectiveness"  (Genet,  July,  1970). 


In  most  complex  systems,  a very  few  areas  are  the  source  of  the 
majority  of  system  problems  and  system  costs.  Usually,  the  most  effective 
approach  to  improve  the  performance  of  a complex  system  is  by  concentrating 
attention  on  these  "critical  few".  The  question  is,  "How  can  we  identify 
these  areas?" 

An  efficient  method  to  identify  the  critical  points  in  a complex 
repair  network  involves  the  use  of  a Markov  model  of  flows  through  the  re- 
pair system.  Initial  work  to  apply  this  technique  to  depot  repair  processes 
was  done  by  Peter  S.  Palmer  (1970)  of  Charles  Stark  Draper  Lab  and  by  Russell 
M.  Genet  (Oct.,  1970)  of  the  Aerospace  Guidance  and  Metrology  Center.  Appli- 
cation of  this  approach  to  the  KT-73  IMU  repair  process  is  reported  by  Watson 
and  Waterman  (1974).  Iwerson,  Berry,  and  Brawner  (1975)  then  extended  the 
Watson-Waterman  model  to  provide  a more  detailed  description  of  the  KT-73 
repair  system.  The  Markov  modeling  technique  has  also  been  applied  by 
Genet,  Martin,  and  Besteda  (May,  1973)  to  depot  repair  processes  for  the 
G-200  Gyroscope  and  for  the  LN-12  inertial  platform. 

The  objective  of  the  above  models  is  to  describe  the  primary  flow 
and  cost  characteristics  of  the  repair  process,  including  "feed  back"  charac- 
teristics. The  value  of  making  specific  changes  in  the  system  may  then  be 
determined  by  resolving  the  model  using  changed  parameters,  and  comparing 
the  average  repair  costs  under  the  old  and  proposed  systems. 
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The  basic  mathematical  structure  of  these  models  is  as  follows: 
Let 


ij 


The  values  for  II 


The  average  number  of  times  that  an  IMU  passes 
through  repair  stage  j 
Cost  of  passing  through  repair  stage  j 
Probability  that  an  item  progresses  from  repair 
stage  i to  repair  stage  j 
j are  the  solution  to  the  system  of  equations 

n.  - 1 


and 


n. 


j 


z 

i 


n *p 

i U 


j = 2,3 ,N 


(90) 


(91) 


where  II  represents  the  receiving  stage  for  the  process.  Equation  (90)  simply 
states  that  one  unit  enters  the  repair  process,  while  equation  (91)  states 
that  the  average  number  of  times  an  item  passes  through  stage  j equals  the 
sum,  over  all  repair  stages  i,  of  the  number  of  times  an  item  passes 
through  stage  i times  the  probability  the  item  progresses  from  stage  i to 
stage  j . 


The  average  cost  (C)  to  repair  a single  unit  is  then 

c - ^ («) 

For  equations  (90)  and  (91)  to  be  valid,  we  must  assume  that  the 
transition  probabilities  P^  are  dependent  upon  the  current  repair  stage  i, 
and  do  not  depend  upon  the  previous  repair  history  of  the  item.  This  is  known 
as  the  "Markov  Property"  assumption  (Vagner,  1969,  p.  740). 

Let  us  now  examine  how  this  model  may  be  used  to  Identify  critical 
areas  in  the  KT-73  repair  process. 

Figure  19  . illustrates  the  Watson-Waterman  Model  for  the  KT-73 
repair  process,  with  all  feedback  paths  identified.  In  the  figures,  each 
repair  stage  is  identified  by  a single  letter  and  each  flow  path  is  represented 
by  two  letters.  For  example,  the  flow  path  AC  represents  flow  from  the  initial 
test  (stage  A)  to  the  Remove  and  Replace  Glmbol  Cluster  Assembly  stage  (stage  C). 
Transition  probabilities  for  each  flow  stage  are  shown  in  Figure  20. 

Since  there  are  N*20  stages,  in  Figure  19,  equations  (90)  and  (91)  describing 
Figure  19  will  consist  of  20  equations.  For  example,  the  first  three  of  thes » 
are 
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FIGURE  19.  KT-73  REPAIR  PROCESS  SHOWING  ALL  FEEDBACK  PATHS 
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where  P^  denotes  the  probability  that  an  item  at  stage  A will  be  routed 
to  stage  B,  and  other  P^  values  are  similarly  defined. 

This  model  may  be  used  to  identify  critical  cost  areas  by  per- 
forming the  following  steps. 

a.  First,  solve  the  system  of  equations  (90)  and  (91)  for  the 

values  of  11^.  Since  (90)  and  (91)  is  a system  of  N linear 
equations  in  N unknowns,  the  values  of  may  be  easily 

determined  using  readily  available  computer  programs. 

b.  Given  11^,  the  cost  of  stage  j repair  for  an  average  (CS^) 


item  is  then  CS, 


C n and  the  cost  to  repair  an  average 


item  is  given  by  (92).  Repair  stages  for  which  CS^  is  large 
represent  high  Impact  stages;  if  the  cost  of  such  stages 
can  be  reduced,  significant  savings  will  be  produced  for  the 
whole  system. 

c.  Finally,  investigate  the  impact  of  specific  feedback  components 
upon  the  average  cost  of  repair.  To  do  this,  select  a spe- 
cific stage  for  further  study.  Suppose  we  select  stage  0, 
the  A.T.P.  Final  Test  stage.  Next,  reduce  the  feedback  transi- 
tion probabilities  by  a reasonable  amount,  say  10Z,  with  a 
corresponding  increase  in  the  feed-forward  path.  Since  we  have 
selected  stage  0,  we  would  reduce  the  transition  probabilities 
for  the  feedback  paths  OB,  0C,  0D,  OF,  and  OH,  and  corres- 
pondingly increase  the  transition  probability  for  path  0T. 

The  system  (90)  and  (91)  would  then  be  solved  using  the  new  co- 
efficients, and  the  average  repair  cost  (92)  for  the  modified 
system  would  be  determined.  The  difference  between  this  cost 
and  the  cost  found  in  step  b represents  the  marginal  savings 
that  can  be  achieved  by  reducing  feedback  at  stage  0.  a 
similar  analysis  would  then  be  performed  for  each  repair  stage 
in  the  network. 
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* 
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For  a detailed  illustration  of  this  analysis  approach,  see 
Watson  and  Waterman  (1974)  or  Iwerson,  Berry,  and  Bravmer  (1975). 

The  above  analysis  process  provides  two  useful  kinds  of  infor- 
mation. Step  b identifies  high  cost  repair  activities.  Small  percentage 
savings  in  the  cost  of  these  activities  should  produce  significant  dollar 
savings  to  the  system  as  a whole. 

On  the  other  hand,  step  c identifies  high  cost  error  flows.  Feed- 
back results  from  one  of  two  types  of  errors;  errors  due  to  improper  repair 
at  some  stage  or  errors  due  to  incorrect  test  or  diagnosis  decisions.  Step 
c analysis  identifies  areas  in  which  small  percentage  reductions  in  feed- 
back flow  may  produce  significant  cost  savings.  Unfortunately,  the  analysis 
does  not  determine  the  source  or  cause  of  these  flows.  Nevertheless,  step  c 


analysis  is  important  in  that  it  identifies  areas  in  which  more  detailed 
analysis  may  be  warranted.  For  these  critical  areas,  the  more  detailed 
analysis  procedures  described  elsewhere  in  this  report  should  be  used. 
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SOME  ASPECTS  OF  STATISTICAL  CLASSIFICATION  METHODS 

by 

Jack  C.  Lee 

Department  of  Mathematics 
Wright  State  University 
Dayton,  Ohio 

1.  INTRODUCTION 

Classification  may  be  considered  as  a problem  of  statistical  decision 
functions.  Suppose  an  individual  is  an  observation  from  one  of  the  k 
populations  ttj. — » tt^-  The  classification  of  an  observation  depends 
on  the  Vector  measurement  X'  = ( — , Xp)  of  that  individual.  An 

optimal  classification  procedures  is  the  one  which  minimizes  the  expected 
cost  or  the  probability  of  misclassification. 

If  q.j  is  the  prior  probability  of  drawing  an  observation  from 
population  tt^  with  probability  density  p^x)  (i=l,  k)  and  if  the 
cost  of  misclassifying  an  observation  from  tt..  as  from  it.  is  C(j/i), 
then  the  regions  of  classification,  Rls  — , R^  that  minimizes  the 
expected  cost  are  defined  by  assigning  x to  Rm  if 

l 9^  Pi  (0  C (m|i)  < l q.  p.  (x)  C(  j | i ) ( j=l , — , k;  j=m), 

i=l  11  i=l  1 1 ~ 

i^m 

See  Anderson  (1958).  We  note  in  passing  that  when  k=2,  then  we  assign 
x to  R,  if  q2p2(x)  C(l.|2)  < qiPi(s)  C(2fl). 

Statistical  Classification  methods  can  also  be  applied  to  medical 
diagnosis,  as  in  this  situation  we  have  essentially  several  possible  popu- 
lations for  any  patient  to  be  classified,  see,  for  example.  Cornfield, 
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et  al  (1973)  or  Anderson  (1973).  With  the  help  of  computer,  statistical 
classification  methods  should  play  more  important  role  in  the 
field  of  medical  diagnosis  for  years  to  come. 


In  the  next  few  sections  we  will  discuss  problems  of  dimensionality, 
multivariate  normal  classification,  equal-mean  classification  and  growth 
curve  classification.  We  will  restrict  our  attention  to  the  situation 
where  normality  is  assumed. 


2.  REDUCTION  OF  DIMENSIONALITY 


It  is  sometimes  of  interest  to  determine  whether  we  can  reduce  the 
dimensionality  of  observations  from  p variables  to  r (<p)  variables 
and  preserve  as  much  as  possible  the  allocation  in  r-dimensions.  This 
is,  of  course,  the  problem  of  principal  component. 

Let  Xi,  — , X^  be  N independent  and  identically  distributed 

p-variate  normal  with  mean  vector  y and  Covariance  Matrix  ].  Also 

N 

let  lx>~ - > i p be  the  eigenvalues  of  S=(sflb)  = l (]L  -X)  (X.  -X)'  , 
N.  i = l 

NX  = J X. , whereas  \x  > — >xp  are  the  latent  roots  of  J\  One 

possible  statistic  to  use  for  the  reduction  of  dimensionality  is 

R = ^ ^ / \ ^i  which  was  Pr°P°sed  by  Rao  (1964).  We  will  state  a 

i=l  ' i=l 

recent  result  of  Krishnaiah  and  Lee  (1976)  which  includes  R as  a 
special  case. 


(2.1)  Tg  = fiT\ ^ Cg j dgh  ln)  - Cgh  A.  (J^  dgh  Afl) 

for  g=l , 2,  — , q,  and  Cgh,  dgh  are  known  constants.  We  note  that  if 
q=l , dgb=l  f°r  a^ 1 b. 


r 

P 


r sfl  for  h = 1,2,  — , 
ugh  lo  for  h = r+1 , — , 

then  we  have  essentially  the  statistic  R proposed  by  Rao. 

The  joint  density  of  Ti , T2 , — , Tq  is 


(2.2)  f(Ti,  — , T ) = N(T;  B 3 1 ) [1  + 7“  { 5 l [ A*  *h  H (T) 

q /n  h=l  k/h  j=l  Jn  nK  n K J 

-2  j,  j,  ",  <D 

^ J,  j jjl  Jl  ^ AJ  .h  AjJh  Aj.h  "J.jjj,111 


p q 


V l 1 1 1 A*  . A*  . b*  • H.  . . (T) 

1=1  h"*l  ox=l  j2=l  j3=l  h h Jlh  Jzh  hh  ,j3  JiJzJ3 


+ 0(n'1)] 


Where 


r 

83  = 2 h=l  Ah  Ah  Ah  ’ Ah  = ^Ai  Aih'  Aq  Aqh^  ~ = ^Tt>  V 
Ag1=  Jl  dghV  Agh  = Ag  Agh’  Ahk  = Ah  " Ak’  bgj  = Aj  Agj  dgj, 

bgf,j  ‘ Aj  Ajg  djf’  N(x;n)  = (2ir)"P/2  laf%  exP  C-  * 5'  ^_1  *]» 


i.e.  the  density  of  N(0,ft),  and  H.  , — , • (x)  = 

J 1 J Q 


(-DS  3S 

N(x;fi)  9xi  — 3x^ 

J J c 


N (x;n) 


where  j,,  j2,  — - , j$  are  s integers  such  that  1 S j^  5 p. 

H.  , (x)  is  the  multivariate  Hermite  Polynomials,  see  Appel 

J Jg 

and  Kampe  de  Feriet  (1926),  and  Khatri  and  Mitra  (1969). 


v i m&ms. 


i mum 


With  (2.2)  we  will  be  able  to  test  various  hypotheses  and  obtain 
confidence  intervals  (regions)  for  functions  of  eigenvalues  \it  — -,Xp. 
In  particular,  the  hypothesis  testing  as  well  as  confidence  interval  for 

r p 

l A.  ) A.  can  be  obtained.  Results  for  the  joint  distribution  of 

i=l  1 j=l  J 

linear  combinations  of  eigenvalues  and  those  of  the  corresponding  complex 
multivariate  normal  case  are  also  given  by  Krishnaiah  and  Lee  (1976). 

In  order  to  make  practical  use  of  (2.2)  and  the  other  results  derived 
by  Krishnaiah  and  Lee  (1976),  estimates  would  have  to  be  substituted  for 
the  A.j , which  was  also  the  situation  reported  by  Lawley  (1956)  in  con- 
sidering tests  for  the  equality  of  last  few  latent  roots.  Certainly 
better  results  would  be  the  ones  free  from  any  unknown  parameters.  Un- 
fortunately they  are  not  yet  available. 


3.  MULTIVARIATE  NORMAL  CLASSIFICATION 


Suppose  there  are  k p- variate  normal  populations  ttu } tt^  with 

mean  vectors  yh s yk  and  covar^.nce  matrices  ] t > One  is 

interested  in  classifying  a new  p-variate  observation  z to  one  of 

these  possible  populations  in  some  optimal  fashion.  Assuming  z has 

k 

prior  probability  qi  of  belonging  to  , l q.  = 1,  then  the  optimal 
classification  method  with  regard  to  posterior  probability  of  correct 
classification  is  to  allocate  z to  that  tt^  for  which 


(3.1)  A.(p)  = In  q.  - klo  |£.  | - ^(z-y.)^"1  (z-y.) 

is  a maximum.  Here,  of  course,  the  parameters  ^ , y^  and  q^  are  assumed 
known. 
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Gelsser  (1964,  1966)  proposed  the  following  Bayesian  method.  Suppose 
tk  has  density  fi(*le1-»^)  where  0^  is  the  set  of  unknown  parameters 


and  i|i.  is  the  set  of  known  parameters;  X.  are  the  data  obtained  from 

I 1 

u.  based  on  N.  independent  vector  observations.  Further,  let 
1 k 1 k 

0 = U 0. , tp  = U to. , g ( 0 1 1^)  be  the  joint  prior  density  of  0,  and 
i=l  i=l  1 

L(X.  1 0 - , <|>.)  be  the  likelihood  fo  the  sample  obtained  from  it.  with  joint 

III 

likelihood. 


k 

(3.2)  L(X|0,iJ/)  = n L(X.  1 0.  ,tp, ) 

i=l  1 1 1 


where  X represents  the  totality  of  samples  X^.  The  posterior 

probability  that  z belongs  to  tt.  is 


(3.3)  Pr(zeiTi  |X,^,q)  q.f  2(z  |X,^,tt1  ) 


where  q"  = (qn  qfc)  and 

(3.4)  f2(z|X,^,TTi)  = / f1(z|0i,ipi,Tri)  P(0i  |X,t|>)  d0.. 


(3.5)  P(01|X,i|»)  =/  P i ( 0 1 X , t^»)  d©9  - / L(X 1 0,ip)  g(0|ip)  d0^ 


where  09  is  the  complement  of  0. , ©9  U0n-  = 0.  It  is  to  be  noted  that 

^2(z|X,  ili,  tk)  is  the  predictive  density  of  z and  P( 0 . | X ,\p)  is  the 
posterior  density  of  0. . For  classification  purposes,  we  may  choose  to 
assign  z to  that  tt.  for  which  (3.3)  is  a maximum.  We  thus  see  that 
all  we  need  to  derive  is  the  predictive  density  of  z.  When  normality  and 
parameters  are  assumed,  then  (3.3)  and  (3.1)  are  equivalent.  We  also  note  that 
in  (3.5)  relevant  constant  of  proportionality  has  to  be  recovered,  otherwise  it 
might  be  too  complicated  to  recover  in  the  final  result,  see  Lee  (1975a). 
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Geisser  (1964),  using  noninformati ve  prior  for  unknown  parameters 
derived  predictive  densities  for  various  combinations  of  mean  vectors  and 
covariance  matrices.  For  the  rest  of  this  paper  we  will  discuss  equal- 
mean  classification  and  that  of  growth  curves. 


4.  EQUAL-MEAN  CLASSIFICATION 


The  problem  of  statistical  discrimination  with  respect  to  different 
unknown  covariance  matrices  and  common  unknown  mean  vector  of  two 
p-variate  normal  populations  tTj,  tt2  was  studied  by  Okamoto  (1961), 
Geisser  (1964)  and  Lee  (1975a). 


Let  y be  the  common  unknown  mean  vector  and  Y , Y the  two 
unknown  covariance  matrices.  By  using  a convenient  joint  prior 

(p+D/2  , (p+l)/2 

s(u.  I, . I ) * II,  I ir'i 
~ 1 2 1 2 

Lee  (1975a)  derived  the  predictive  density 

(4.1)  f3(z|X,  it.)  « r[(N.+l-p)/2)  r[Nj/2]  lAiz'  Nl/2  'Aj'  (Nj 

• I t(N.-l)/2  B-(N+l-p)/2  jn^a^  + t(N.+l)Aj  dt, 

i t j,  where  NiX1  « Xid.  A1  = (Xid  -X.)(X.d  -X.)', 

N. 

Aiz  = Ai  + nT+T  (5-Xi)(?-?i)  ’ H * (Ni+1 )“ 1 (N.X.  + z), 

Bz  = 1 + t[l+Nj(N.+l)(y-Xj)'{NjA.2n(Ni+l)Aj}(y-Xj)] 

and  X.  X.M  are  N.  independent  observations  from  it..  It  is  to 

be  noted  that  the  proportionality  constant  is  irrelevant  for  our  purpose. 
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A computer  program  written  in  Fortran  was  developed  for  the  evalua- 
tion of  the  above  integral.  We  also  note  that  in  the  integral  we  have 
for  any  t* 

co 

(4.2)  / h(t)  dt  < 2N:p/2  (N.-p)-1  t*‘(Nj-P)/2  |A.  I"3*  = Ej 

where  h(t)  is  the  integrand  of  (4.1).  Thus  the  integral  in  (4.1)  can  be 

considered  as  a finite  integral  from  0 to  t*  with  error  bound  Ej.  The  com- 
puter program  also  provides  error  bound  E2  of  integration  from  0 to  t*. 
Hence  the  error  bound  of  the  integral  in  (4.1)  is  simply  Ei+  E2. 


5.  GROWTH  CURVE  CLASSIFICATION 


The  growth  curve  model  introduced  by  Potthoff  and  Roy  (1964)  is 


(5.1)  E(Xp  x m)  Bp  x m Tm  x r Ar  x N 


where  t is  unknown,  B and  A are  known  design  matrices  of  ranks 
m < p and  r < N respectively.  Further,  the  columns  of  X are 
independent  p-dimensional  normal  variates  having  a common  unknown  co- 
variance  matrix  J,  i.e.  G(X|t,£)  = N(-;BtA,  J8In)  where  8 denotes 
the  Kronecker  product  and  G(*)  is  the  c.d.f. 


We  will  consider  the  situation  where  we  have  observed  k growth 
curves 


(5.2)  G(Xj  |tj  ,ir^)  = N(*;  Bt^A^  , 1=1»  — * K and  a future 

observation  z,  of  dimension  p * 1,  which  is  known  to  be  drawn  from  one 
of  the  K populations  tTj,  — , with  known  prior  probabilities 

V V We  assume  G(?lTi’Ii,  V = N(*i  BriFi»  li)  where  Fi  is  a 

known  vector. 
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By  a convenient  join  prior 

(5.3)  g(t„  — t r,‘.  — . iV)  - * ili  (ptl,/2 

K k QFl  “ 

the  predictive  density  of  z was  shown  by  Lee  (1975b)  to  be 

(5.4)  f4(z|X.,  rr.)  <*  C*  |B 'ST  XB  |'(Ni_r)/2  |G.  |m/2  |Z'X.XrZ|(Ni'm)/2 

* | (B'ST 1B)~  1 + B*(z-BQi)  G.(z-BQi)'  B*'|'(Ni+1'r)/2 

•|Z'(X.xr  = zz')  z|"(Ni+1"m)/2 

for  i = 1 k 

» » 

where 

Gi 1 = MT  1 + (z -V.)'  Z(Z'S.Z)'1  Z'(z-V.) 

Mi  = 1'Fi^HiHi)'1  <V  ^ " <V  F-j ) 

Vi  = X.Ar  (A.Ap'1  F.,  B*  = (B'B)’V 

(5.5)  Qi  = B*^.  + B*S.Z(Z'S.Z)"1  Z'(z-V.) 

Si  - x.d-A^A.Arr1  a.)  xr 
rWN.+l-r)]  r[%(N.+l-m)] 

Q*  _ I I 

1 r[%(N.+l-r-m)]  r[%(N.+l-p)] 

and  Z is  p x p-m  and  of  rank  p-m  such  that  B'Z  = 0. 

The  constant  of  proportionality  is  irrelevant  and  will  be  absorbed  by 
the  proportionality  sign  <*. 

The  predictive  densities  for  various  combinations  of  parameters 
together  with  a special  covariance  structure  were  given  by  Lee  (1975b). 

In  case  locational  structure  is  present,  the  growth  curve  classification 
is  expected  to  yield  result  better  than  the  general  multivariate  normal 
classification. 
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DIAGNOSIS  OF  RELIABILITY  REPAIR 


STAGE  AND  REMEDIES 
Edward  Bilikam  and  Albert  H.  Moore 
Abstract 

There  are  three  stages  of  reliability  failure  patterns, 
(1)  infantile  failure,  (2)  cyclic  failure  and  (3)  fatigue 
failure.  These  stages  determine  the  general  method  and 
condition  of  the  replaceable  units.  Some  remedies  are 
burn-in,  quality  control,  parts  control  and  scheduled 
maintenance  (overhauls).  These  measures  can  be  applied 
according  to  the  stage  of  the  reliability  failure  pattern. 
The  failure  patterns  should  be  determined  by  reliability 
testing  and  subsequent  analysis.  In  the  analysis  of  the 
data  it  is  necessary  to  determine  what  stage  you  are  in. 

One  way  of  doing  this  is  to  use  a Likelihood-Ratio  Test. 

Such  a test  is  presented  with  discussion  of  the  power  of 
the  test.  This  test  also  has  applications  to  reliability 
growth . 


202 


INTRODUCTION 


We  will  be  interested  in  the  failure  of  equipment 
which  will  in  general  be  replaced  with  repaired  equipment. 

The  fact  that  the  equipment  is  old  enough  to  have  been  re- 
paired means  that  we  are  not  just  considering  initial  per- 
formance reliability  but  reliability  of  maintained  equipment. 

A way  of  characterizing  the  reliability  of  maintained 
equipment  is  the  hazard  rate  curve.  Hazard  rate  curves  can 
be  classified  into  three  general  patterns.  The  first 
pattern  is  that  of  infantile  failure . In  this  stage  the 
failure  rate  is  much  higher  than  after  the  failure  rate  has 
stabilized  and  is  characterized  by  a sharply  decreasing 
failure  rate.  Another  pattern  is  characterized  by  eye  lie 
f ai  lure . The  installed  equipment  in  this  phase  has  an 
operating  time  before  failure  about  equal  to  the  MTBF  and 
varies  randomly  on  either  side.  The  cyclic  time  of  replace- 
ment is  very  nearly  the  MTBF.  The  stage  is  also  character- 
ized by  a slowly  increasing  failure  rate.  The  number  of 
replacements  in  a fixed  unit  of  time  is  the  time  divided 
by  the  MTBF.  The  third  phase  of  the  failure  pattern  is 
the  fatigue  phase . Here  the  equipment  operates  until  the 
fatigue  hazard  sets  in  and  the  equipment  exhibits  a sharply 
increasing  failure  rate. 

Remedies  - Infant i 1 e Failure 

The  above  description  of  infantile  failure  leads  to  a 
diagnosis  of  a producibility  problem . Something  must  be 
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done  to  enable  the  operation  of  the  equipment  for  a useful 
period  (see  reference  1).  One  remedy  is  burn-in.  Just  how 
long  the  burn-in  must  be  applied  to  each  equipment  was  dis- 
cussed in  reference  [2].  Burn-in  however  can  be  expensive 
and/or  d#structive.  For  instance  if  it  takes  one  hundred 
equipments  to  obtain  one  equipment  that  will  survive  to  the 
operation  stage  then  burn-in  is  very  expensive  indeed.  It 
is  even  more  critical  if  the  burn-in  leads  to  destructive 
failures.  Then  the  equipment  lost  during  burn-in  just  in- 
creases the  cost  per  item.  This  is  why  burn-in  of  micro- 
components is  usually  feasible  while  burn-in  of  sub-systems 
is  usually  prohibitive. 

On  the  component  level  two  other  remedies  may  be  useful. 
These  are  parts  control  and  quality  control.  The  natural 
laws  of  failure  determine  which  is  more  useful.  Quality 
control  is  applied  to  production  equipment  parts  when  it  is 
determined  that  the  quality  of  the  parts  can  be  maintained 
within  engineering  tolerances  with  only  an  occasional  chance 
defect.  Parts  control  however  means  screening  the  parts. 

That  is  weeding  out  the  bad  parts  in  selective  inspection 
to  obtain  preferred  classes  of  parts  to  use  in  critical 
assemblies.  The  idea  is  hold  equipment  performance  at  a 
minimal  but  necessary  level. 

When  none  of  the  above  remedies  can  alone  improve 
infantile  failure  there  is  a definite  need  for  reliability 
growth.  What  is  needed  is  a reliability  growth  into  the 
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class  of  cyclic  failure  patterns.  Redesign  would  then  be 
a real  consideration. 

Remedies  - Cyclic  Failure 

In  cyclic  failure  the  producibi 1 ity  of  the  equipment 
is  usually  not  in  question.  The  problem,  if  there  is  a 
problem  is  low  MTBF.  In  general  this  calls  for  an  increase 
in  the  scale  parameter  of  the  failure  time  distribution. 
Failures  are  occurring  just  too  soon  after  the  equipment  is 
put  into  operation.  The  solution  is  to  lengthen  the  equip- 
ment life.  Redundancy  and  fault  tolerance  are  considerations 
That  is  one  should  try  to  rearrange  and  work  with  the  base 
equipment  that  is  on  hand.  Software  redundancy  for  proces- 
sors should  be  considered. 

If  it  can  be  shown  that  cyclic  failure  is  present,  the 
next  life  step  is  fatigue.  Actually  fatigue  failure  is 
unwanted.  The  kind  of  repair  for  cyclic  failure  is  un- 
scheduled and  preventive  maintenance.  That  is  "beef-up" 
the  system.  This  may  be  difficult  in  airborne  systems  where 
weight  and  cost  are  critical.  Engineering  design  is  some- 
times load  critical  so  that  when  preventive  maintenance  is 
applied,  the  addition  of  new  components  changes  the  loading 
in  an  inappropriate  way.  The  results  of  "improvements"  must 
be  tested  (reliability  environmental  test)  to  see  the  result 
of  changing  the  system. 

Remed  ies  of  Fatigue 

Some  components  are  always  subject  to  fatigue.  Some 

relays  are  an  example.  There  is  little  choice  of  a remedy 
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for  fatigue-retrofit  and  scheduled  maintenance.  When 
components  are  worn  out  they  must  be  replaced.  When  the 
whole  system  is  in  fatigue,  scheduled  overalls  are  a must. 
Fatigue  is  characterized  by  a gradual  degeneration  until  a 
critical  fatigue  life  where  catastropic  failure  sets  in. 

Identification  of  Life  Stages 

The  stages  of  failure  patterns  discussed  above  are 
characterized  by  the  shape  of  the  failure  probability  density 
function  (p.d.f.).  This  function  formulates  quantitatively 
where  the  failures  are  most  probable,  in  early  failure, 
centralized  failure  or  wear  out  failures.  The  class  of 
Weibull  distributions  with  different  shape  parameters  will 
be  used  to  model  the  failure  distributions.  The  Weibull 
p.d.f.  has  the  form 

f(x)  = (B/0)xB_1  exp  - (xB/0) 

x,  0,  B > 0 (1) 

An  additional  parameter  to  locate  the  distribution  is  ob- 
tained by  the  transformation  x = t - c,  where  no  failures 
occur  with  probability  one  before  c > 0.  Generally  we  will 
assume  c is  known  or  zero,  but  c may  be  estimated  (see 
Harter  and  Moore)  . B is  the  shape  parameter  and  0 is  the  B1*1 
power  of  the  scale  parameter.  Given  the  Weibull  density  (c=0) 
the  reliability  function  (probability  of  survival)  is 

R(t)  = exp  - ( t B / 0 ) (2) 

for  operation  time  t. 
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Each  of  the  parameters  may  be  estimated  from  suitable 
sample  data.  Two  common  types  of  censored  data  treated  in 


the  literature  are  called  Type  I and  Type  II  censoring. 

Type  II  censoring  occurs  when  n equipments  are  put  on  test 
and  the  test  is  continued  until  r items  have  failed  (r  s n) . 

In  Type  I censoring  the  test  is  terminated  at  a fixed  time 
regardless  of  the  number  of  items  that  have  failed.  Harter 
and  Moore  [4]  using  the  method  of  maximum- likelihood  have 
derived  simultaneous  nonlinear  equations  for  the  parameters 
of  two  and  three  parameter  Weibull  distribution  and  iterative 
techniques  for  their  solution.  Wingo  [7]  presented  a way 
of  accelerating  the  solution  of  the  simultaneous  equations 
using  a gradient  technique.  For  multiple  independent  Type  I 
censored  samples  with  failure  times  known  and  unknown  Bilikam 
and  Moore  have  used  maximum  likelihood  techniques  and  the 
gradient  iterative  method  to  solve  for  0 and  B.  Therefore 
estimation  techniques  exist  to  estimate  the  parameters  of 
the  Weibull  distribution  for  type  of  data  considered  above. 

The  stages  of  the  failure  patterns  discussed  above  may 
be  more  precisely  classified  by  the  shape  parameter  of  the 
Weibull  distribution  as  follows: 

a.  0 < B < 1.0  infantile  failure  (see  Figure  1) 

b.  ljO  < B < 3.6  gradual  progression  toward  cyclic  failure 

c.  3.6  < B < 4.0  cyclic  failure  (see  Figure  2) 

d.  B > 5 fatigue  failure  (see  Figure  3) 

An  estimation  of  the  failure  pattern  within  the  scope 
of  parametric  statistics  is  possible  by  estimating  the  shape 
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parameter.  The  estimated  variance,  bias,  and  confidence 
intervals  for  estimates  will  guard  against  reaching  eroneous 
conclusions  based  on  small  samples.  In  addition  one  must  be 
sure  the  testing  situation  matches  the  assumed  conditions. 

'l 

Fatigue  in  materials  testing  is  well  known,  and  computers 
have  been  found  to  operate  (CPU  up  and  downs)  in  the  near 
cyclic  stage  (1.12  < B < 1.62).  An  airborne  equipment  test 
showed  a B of  2.22  that  is  the  middle  ground  of  cyclic 
failure.  An  infantile  pattern  of  B = 0.32  (reference  [3]) 
and  a Radar  B = 0.72  are  known. 

Within  Stage  Rel iabi 1 ity  Growth 

With  any  reliability  improvement  program  the  results 
should  be  judged  for  reliability  growth.  Reliability  growth 
can  occur  within  a failure  stage,  not  infantile  failure 
stage,  but  cyclic  and  fatigue.  In  the  next  section  change 
from  one  stage  to  another  will  be  examined.  This  section  is 
concerned  with  changes  (improvements)  of  scale  given  that 
the  equipment  does  not  change  in  the  shape  of  its  failure 
pattern.  The  test,  a statistical  hypothesis  test,  concerns 
k samples  (test  observations)  with  samples  of  varying  size, 
all  from  an  underlying  Weibull  distribution  with  the  same 
shape,  but  possibly  different  scale  parameters.  The  test 
is  known  as  a k-sample  one  way  classification  test  (analysis 
of  variance  in  the  Normal  Law  assumption) . For  the  k > 2 
samples  the  null  hypothesis 
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Hq;  n = (the  k scale  parameters 

ei1/B)  for  i - 1,2, ...  ,k.  (3) 

That  is,  it  is  tested  that,  throughout  the  k reliability 
improvement  tests  there  was  no  improvement  in  the  scale. 
McCool  [5]  is  a study  of  the  statistical  test  applications 
with  tabled  results  in  which  Type  II  censored  samples  are 
allowed.  Of  course,  for  reliability  growth,  there  should 
be  a significant  deviation  from  hypothesis  (3) . Now  the 
MTBF  for  a Weibull  population  is 

MTBFi  = niT(l/B  + 1) 

i = 1,2,. ...k  (4) 


where 


(5) 


Hence  testing  that  the  n ' s are  all  equal  from  sample  to 
sample,  given  the  shape  B does  not  change,  tests  that  the 
MTBF's  are  not  improving.  The  statistic  used  is  a F ratio 
of  the  within  group  shape  estimates  to  the  pooled  shape 
estimate . 


Change  of  Fai lure  Pattern 

As  was  discussed  above,  the  consequences  of  infantile 
failure  are  so  severe  that  it  is  wished  to  know  when  this 
stage  has  been  changed  to  cyclic  failure,  or  normal  mainte- 
nance conditions.  Then  too,  for  purposes  of  planning  over- 
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hauls  and  retrofits,  a change  from  cyclic  failure  to  fatigue 
failure  should  be  detected.  These  changes  of  stage  must  be 
observed  from  reliability  test  results.  The  two  sample 
change  of  shape  parameter  is  derived  from  the  likelihood 
equations. 

The  null  hypothesis  is  that  there  is  no  change  of  shape 
parameter  B from  the  first  sample  (size  n^  to  that  of  the 
second  sample  (size  n2)  after  possible  improvement  or  change 
of  shape  due  to  overall  effect  of  age. 

Hq:B1  = B2  (6a) 

The  alternative  hypothesis  is  unequal  shape  parameters 

H j : Bx  t B2  (6b) 

The  test  is  independent  of  the  scale  parameter.  The 

maximum  likelihood  ratio  statistic  is  to  be  calculated  from 

observed  failure  times  of  the  first  sample  xi»*.*»xn  and 

^ 1 

those  of  the  second  sample  y.,...,y  . X was  derived  for 

n2 

the  test  by  J.  E.  Bilikam. 


A = (B^Bj) 


where  B ^ , 6^  ©2 
are  genera  1 M . L . 


n * ^ n 

(b1/b2) 


i = l 


A ft  ^ 1 A ft  H - 

(e  j/e  x)  (02/e2) 


are  M.L.  estimates  under  HQ 
estimates  for  each  sample. 


(7) 

AAA 

and  Bj,  B2>  0j, 
Under  : 


* 


213 


♦ 


(8) 


n.+n,  nl  E In  x.  n2  B In  y. 

L ln  -i  -H-  * A.  ln  V-*-1 


Bj  i = l 


= 0 


0I  j=1 


0. 


and 


1 B 


2 B. 


A u I a c l 

, = I x /n  , 0,  = l y.  /n 
1 i= 1 1 1 1 j=l  \ 


Under  h^: 


and 


n, 


ni/Bi  + l.  ln  xi  - .1.  xi  — r1  = 0 
1=1  1 = 1 6 


n, 


1 B 
0 = l x /n 

1 i= 1 1 1 


(9) 


n2/B2 


n2  n2  B In  y 

I y a2  - o 


♦ I in  y.  - I 
j=l  J j=l 


J 


e. 


and 


Jl  . 

« 2 B _ 

e2  - lmi  yj2/nr 


(10) 


Each  set  of  the  above  equations  (8),  (9),  and  (10)  are  solved 
simultaneously  to  obtain  MLE's.  Equation  (7)  X = L(to)/L(fi) 
is  the  value  of  the  maximum  likelihood  ratio,  the  distribution 
of  which  may  be  used  to  construct  the  power  of  this  test. 

Under  , the  critical  points  of  the  test  are  obtained  by 
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Petrick  [6].  It  was  also  shown  by  Petrick  that  the  critical 
values  X (a,n,,n_)  are  not  a function  of  B.  , 0,,  0_  but  only 

C l £ lit 

of  the  sample  sizes  n^ , and  the  significance  level  a. 

Some  values  of  the .critical  value  follow  in  Table  I. 

Table  I 


Sample  Sizes 

Significance 

Critical  Point 

n. 

n„ 

a 

X 

1 

2 

c 

10 

10 

0.05 

0.1108 

10 

20 

0.05 

0.1154 

20 

20 

0.05 

0.1277 

30 

30 

0.05 

0.1303 

10 

20 

0.01 

0.0230 

20 

20 

0.01 

0.0264 

30 

30 

0.01 

0.0296 

When  X < 

X ^ the  hypothesis  H^: 

is 

rejected.  The 

power  of 

the  test, 

the  probability  of 

not 

making  a type  II 

error  is 

then  the 

probability  of  rejecting 

H^:  Bj  = B2  when 

indeed  , 

H.  is  true 

; that  is  when  there 

i s 

a shape  change. 
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The  power  was  obtained  by  Monte  Carlo  simulation.  The  simu- 
lation required  solution  of  (9)  and  (10),  for  equation  (7) 
for  each  trial  assuming  values  of  and  B^- 

To  generate  the  random  Weibull  numbers  the  probability 
integral  theorem  implies  a random  number  r on  (0,1)  is 
related  to  the  Weibull  number  x, 

x = -n [ In ( 1-r) ] 1/8  (11) 


Substituting  x^  and  y ^ in  (7)  with  n ^ and  n2  respectively 
from  (11)  it  is  clear  that  X does  not  depend  on  6^  and  6 2 
(i.e.,  on  and  n2).  Hence  the  distribution  of  X is 

independent  of  the  scale  parameters.  Hence  the  power  is 
independent  of  the  scale  parameters.  Actually  Petrick  shows 
this  only  under  H^B^  = but  it  is  easily  seen  to  follow 
under  Hj  : Bj  / B£ • 


The  power  of  the  test  was  analyzed  from  Monte  Carlo 
simulation  results  and  is  tabled  on  the  following  page. 

Summary  and  Cone  lusions 

From  the  failure  times  observed  during  reliability 
tests  and  the  scale  and  shape  parameter  estimates  one  can 
determine  the  life  stage  a repairable  equipment  is  in. 

Several  remedies  for  infantile  failure  and  fatigue 
failure  are  possible  but  not  always  applicable.  Infantile 
failure  usually  indicates  a pr oduc ib i 1 i t y problem  that 
usually  calls  for  engineering  changes  and/or  design  fixes. 
Cyclic  failure  requires  unscheduled  maintenance.  Fatigue 
failure  requires  overhauls  and  scheduled  maintenance. 
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Table  II 


Power  of  the  Two  Samp  1 e Test  of  Weibu 1 1 
Change  in  Shape  Parameter 

H0:B1  = B 2 vs  ‘ H1  : B1  **  B2 
a - 0.05 


Sample  Sizes  Shape  Parameters 


nl 

n2 

B1 

B2 

Simulation 

Trials 

Power 
P (re j ect 

10 

20 

0.5 

1 . 0 

1000 

0.531* 

10 

20 

0.5 

2 . 5 

1000 

0.994 

10 

20 

1.0 

2.5 

1000 

0.768 

10 

20 

2 . 0 

4.0 

1000 

0.531* 

10 

20 

3.5 

6.0 

1000 

0.330 

20 

20 

0.5 

1.0 

500 

0 .802* 

20 

20 

1.0 

2.0 

500 

0.802 

20 

20 

2.0 

4.0 

500 

0.802 

20 

20 

4.0 

8 . 0 

500 

0.802 

20 

20 

8 . 0 

16.0 

500 

0.802 

30 

30 

0.5 

1 . 0 

500 

0.942* 

30 

30 

1.0 

2.0 

500 

0.942 

30 

30 

2.0 

4.0 

500 

0.942 

30 

30 

4.0 

8.0 

500 

0.942 

30 

30 

8.0 

16.0 

500 

0.942 

•Note:  When  B./B  is  constant  and  a level  constant,  then  the 

A fc  A 

power  will  be  the  same,  because  X values  will  be 
generated  by  the  same  numbers. 
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Reliability  growth  can  be  planned  by  a reliability 
improvement  program.  For  equipments  exhibiting  infantile 
failure  the  statistical  test  would  be  change  in  shape  parameter 
from  0 < Bj  < 1.0  t;o  B2  > 1.0.  For  fatigue  and  cyclic  failures 
one  can  improve  the  equipment  by  increasing  the  scale 
parameter  from  0^  to  62  where  0^^  < 02  assuming  the  shape 


parameter 

is  constant 

B1  " B2  * 

Another 

approach  is 

a change 

in  shape 

parameter  B^ 

to  B 2 where  B ^ > B 

2 and  in 

fatigue 

region . 

Improvement 

Decisions 

Condition 

Change 

Shape 

Scale 

Infantile 

Increase 

B 0 

< B.  < 1 

to  B.  > 1 

ei  , e 

Fai lure 

1 

L 

1 2 

Cyc 1 ic 

Increase 

0 

B1  = 

B 2 

O 

A 

CD 

Failure 

(MTBF) 

L 

1 

Fatigue 

Decrease 

B 

Bi  > 

B2 

CD 

K-* 

A 

CD 

Increase 

0 

Increase 

0 

11 

H 

CO 

B2 

0X  < 0 

The  risks  involved  with  the  decisions  table  are  two  fold,  (1) 
risk  of  type  I error  is  a and  (2)  risk  of  a type  II  error  is 
1.0  - power. 

Since  0j,  Bj>  B2  are  un^nown  parameters  the  actual 

value  is  determined  within  intervals  by  confidence  interval 
estimates.  But  changes  from  0^  to  may  be  tested  with  the 
McCool  test  and  changes  from  to  B2  may  be  tested  with  the 
maximum  likelihood  ratio  test  of  the  last  section.  The  power 
the  maximum  likelihood  ratio  test  seems  to  be  high  in  the 
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Appendix 


Reliability  Growth  by  1C  Steps 

The  two  sample  change  of  shape  parameter  was  discussed 

\ 

in  the  text  above.  However,  the  improvement  may  involve 
more  effort  than  one  reliability  fix  and  test  cycle. 

Suppose  a set  of  K life  tests,  each  obtaining  n^(i  = l,...,k) 
sample  observations.  Then  the  null  hypothesis  to  be  tested 
is 


H0:  Bi  = B2  = •••  = Bk  (Al) 

and  the  alternative  hypothesis  is  that  is  false.  For 
this  test  the  likelihood  ratio  test  can  be  derived  in  a 
similar  way: 

. k „ . n . „ n.  "i  B.-B. 

x = IT  (e./e.)  3(b./b.)  3 IT  x } 3 (A2) 

j=l  3 3 3 i= 1 3 


* J5 

where  B^,  0^  (j  = l,...,k)  are  M.L.  estimators  under  and 

B.,  0.  are  maximum  likelihood  estimates  for  each  sample. 

3 3 

To  obtain  the  MLE  estimates  under 


0 j , 8^  j = 1 , . . . ,k 


the  equations  to  follow  must  be  solved 


n . 
J 


n . / B . ♦ Y In  x..  - y x.3.. 

11  L,  -11  _ L 


3 B4  In  x.  . 


3 J 


i = 1 


ij 


i = l 


-ii-  = 0 


(AS) 
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1 SSB 


1,2,.  . 


0 . 
J 


J U * 

* l xn/ni  for  j 
i = l -1  J 


• , k 


(A4) 


and  to  obtain  the  MLE  estimates  under  H, 


n . a 

j B.  In  x.  . 

-11 


l ni/6l  + l ln  II  x - l l x.1. 
j=l  J 1 j-1  i-1  1J  j=l  i=l 


= 0 


0 . 
J 


( A5 ) 


where 


e . 

j 


1 

l x^/n.  for  j 

j = l 3 3 


1,2, . . . ,k 


(A6) 


It  is  clear  from  (Al)  and  the  argument  discussed  in  the 
text  that  the  critical  points  are  independent  of  B ^ and 
depend  only  on  nJf  n2>  ....  nR  and  a. 

Also  the  Power  of  the  test  is  independent  of  the 
®1»  •••»  Parameters. 
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STAT I bT I CAL  ANALYSIS  AND  EVALUATION  OF  DATA  BASE 


FOR  SIMULATION  STUDIES 

Much  of  the  effort  expended  in  the  field  of  simulation 
has  been  directed  toward  pioblems  of  model  building,  computa- 
tional methods,  and  general  simulation  languages.  Little 
research  has  been  done  on  problems  of  data  base  construction 
for  simulation  studies.  As  shown  in  Figure  1,  data  base 
construction  is  an  indispensable  part  of  a successful  Simula- 
tion  study.  To  a large  degree  model  building  depends  on  the 
data  analysis  in  terms  of  relevancy  and  accuracy  of  assump- 
tions of  the  designated  model.  Although  the  level  and  type  of 
data  base  requirements  may  vary  for  different  simulation 
models,  the  need  for  a sound  data  base  is  the  same.  This 
paper  will  first  discuss  briefly  the  specification  of  data 
requirements,  then  explore  the  data  collection  problem,  and 
finally  suggest  the  statistical  techniques  for  data  analysis 
and  data  validations. 

Specification  of  Data  Requirements 

The  data  requirements  for  simulation  studies  can  be 
grouped  into  two  major  categories:  (1)  historical  data  for 

estimating  the  values  of  parameters,  the  behavior  of  variables, 
and  the  nature  of  relationships;  and  (2)  a data  recording 
system  that  will  capture  data  at  their  source  and  keep  all 
pertinent  data  records  updated.  The  data  recording  system  is 
an  important  part  of  model  building,  particularly  in  simulators 
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designed  for  operational  control  of  operating  systems.  It 
will  not  be  dealt  with  here  in  this  paper. 

The  degree  of  abstraction  of  the  mathematical  model 
representing  the  system  under  study  will  determine  the  histori- 
cal data  base  for  the  simulation  experiment.  The  analyst 
must  determine  and  specify  what  data  are  needed  for  estimating: 
(1)  the  parameters  of  the  system;  (2)  the  behavior  of  the 
variables  of  the  system;  and  (3)  the  nature  of  relationships 
in  the  system. 

A system  parameter,  a quantity  which  affects  the  perform- 
ance of  the  system,  can  be  predicted  with  certainty,  but  either 
cannot  or  will  not  be  altered  by  those  operating  the  system, 
such  as  with  population  parameters  of  random  variables,  cost 
figures,  maximum  queue  length,  storage  limitations,  etc.  The 
behavior  of  the  variables  is  related  to  the  probability  dis- 
tribution of  random  variables.  It  is  sometimes  referred  to 
as  a primary  variable  since  all  these  variables  act  in  an 
independent  manner.  The  nature  of  the  relationship  is  to 
determine  dependencies  among  internal  components,  influences 
of  external  forces,  and  abstract  identities  that  are  included 
for  computational  convenience. 

The  Data  Collection  Problem 

In  constructing  a data  base,  the  problem  of  nonavailability 
of  raw  data  is  a major  constraint  in  a simulation  study. 

Rarely  does  a firm  compile  the  type  of  information  needed  for 
analysis.  For  instance,  consider  the  problems  involved  in 

226 


attempting  to  estimate  the  cost  of  a lost  sale  or  the  cost  of 
a backorder.  Here  one  must  estimate  the  cost  of  intangibles, 
such  as  lost  goodwill  which  may  be  converted  into  the  loss  of 
business  in  the  future.  Thus,  the  problem  of  nonavailability 
of  the  data  is  a major  factor  in  constructing  a viable  data 
base. 

The  second  related  problem  is  that  the  data  being  col- 
lected is  seldom  in  a form  which  is  suitable  for  analysis. 

For  example,  in  undertaking  a freight  study,  the  analyst  needs 
accurate  information  on  the  rate  formulations  in  terms  of 
product,  quantity,  size,  distance,  modes  of  transportation  and 
so  forth.  Unfortunately , the  information  usually  available 
to  him  is  the  rate  structure  which  is  so  complicated  that  it 
is  difficult,  if  not  impossible,  to  analyze. 

Another  problem  which  impairs  construction  of  the  data 
base  is  that  the  data  may  be  obtained  from  various  sources  and 
may  be  inaccurate  and/or  imcomplete.  An  analyst  must  con- 
stantly search  for  more  current  data  as  well  as  for  ways  to 
estimate  from,  and  validate,  available  data. 

The  analyst  is  initially  concerned  with  where  he  can 
begin  to  gather  essential  data,  regardless  of  the  type  of  study 
he  is  conducting.  In  studies  of  business  operating  systems, 
the  company's  sales  invoices  or  bills  of  lading  provide  an 
excellent  starting  point  to  begin  gathering  the  data.  A list 
of  Internal  Sources  of  Data  is  presented  in  Table  1,  and 
External  Sources  of  Orta  in  Table  2. 

227 


, «*•>«*  « sW5 


Table  1 


INTERNAL  SOURCES  OF  DATA 


Source 


Type  of  Data 


Accounting  Records 


Engineering  Records 


Inventory  Records 


Maintenance  Records 


Personnel  Records 


Production  Control 
Records 


Quality  Control 
Records 


Tool  Crib  Records 


Warehouse  Records 


Overhead  charges,  taxes,  utility  costs, 
transportation  costs,  profits,  profit 
margins . 

Material  specf ications , equipment 
performances,  technological  ordering. 

Raw  materials,  in-process,  finished 
goods . 

Reliability  of  components  of  system; 
machine  downtime,  failure  rate,  repair- 
time and  waiting- time  distributions; 
frequencies  of  external  perturbations. 

Salaries,  absenteeism,  medical  records, 
skill  classifications,  labor  availabi- 
lity, labor  turnover. 


Schedule  status,  setup  times,  routing 
and  sequencing  information,  assembly- 
line balancing,  loading,  decision  rules. 

Machine  performance,  scrap,  defectives, 
probability  distributions  of  attributes 
and  measurements,  effect  of  machine  age. 

Frequency  of  demands,  waiting- time  dis- 
tributions, frequency  of  tool  breakage, 
lifetime  distributions  of  tools. 

Frequency  of  issuing  materials  and 
supplies,  peak  periods  of  activity, 
waiting-time  distributions,  frequency  of 
stockouts,  individual  and  aggregate  rate 
of  expenditure  of  materials  and  supplies 


Table  2 


EXTERNAL  SOURCES  OF  DATA 


Customers 

Governmental  agencies 

Industrial  and  trade  organizations 

Insurance  Companies 

Management  consultants 

Market  research  firms 

Publications  in  journals  of 
professional  organizations 

Standard  data 

Universities 

Vendors  and  suppliers 
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Statistical  Techniques  in  Data  Base  Analysis 

1 . Sampling  Considerations 

The  development  of  the  sample  design  is  a critical  stage 
in  the  establishment  of  a data  base.  The  two  most  important 
features  of  a sample  are  its  size  and  the  manner  in  which  it 
was  selected.  Much  of  the  study  of  sampling  statistics  con- 
cerns the  determination  of  these  two  characteristics.  As 
expected,  this  determination  is  based  upon  the  specific  condi- 
tions prescribing  the  purpose  of  the  sample.  For  instance, 
the  sample  in  the  business  system  study  has  to  be  large 
enough  to  be  representative  of  a product  group's  distribution 
pattern  in  each  geographical  subdivision,  yet  small  enough  to 
be  relatively  inexpensive  to  collect  and  prepare  by  the  analyst. 

A simple  random  sampling  may  be  simple  to  use  but  one 
must  carefully  evaluate  the  result.  While  the  population  as 
a whole  may  be  adequately  represented,  the  sample  may  not  be 
representative  of  each  product  group  and  geographical  sub- 
division due  to  the  chance  variation  that  some  subdivisions  may 
have  an  insufficient  number  of  observations  to  provide  adequate 
i nformati on . 

When  many  subdivisions  are  involved  stratified  random 
sampling  should  be  selected.  The  advantages  of  this  procedure 
follow  from  the  fact  that  the  population  is  divided  into  sub- 
divisions. Thus,  only  a relatively  small  number  of  observa- 
tions are  needed  to  determine  the  characteristics  of  each  sub- 
division. 

2 . Sales  Forecasting 

After  the  sample  has  been  designed  and  taken,  the  analyst 
can  proceed  to  sales  forecasting.  There  are  two  levels  of 


sales  forecasting  needed  in  business  system  studies:  One 

level  of  sales  forecasting  is  the  aggregate  sales  forecast, 
such  as  monthly,  quarterly,  or  even  yearly  sales  forecasts 
for  the  product  by  market  regions.  The  second  level  of  fore- 
casting is  detail  forecasting,  which  is  critical  for  a simu- 
lation study,  such  as  frequency  of  order  arrival  and  order 
size  for  each  product  by  market  region.  In  order  to  project 
these  two  level  of  sales  forecasting  trend  and  seasonality  are 
necessary. 

a.  Trend  of  the  Sales.  Most  companies  have  an  extensive 
amount  of  information  on  trends  established  for  their  products 
with  which  the  analyst  can  predict  a sales  forecast.  If  the 
trend  data  are  not  available  or  are  insufficient,  one  can  then 
obtain  the  trend  prediction  by  analyzing  the  historical  sales 
data.  The  simplest  method  to  determine  the  trend  is  that  of 
least  squares. 

The  use  of  a least  squares  method  does  not  eliminate  all 
subjectivity  from  the  analysis  of  time  series.  Among  other 
things,  a decision  must  be  made  on  the  number  of  years  to  use, 
what  adjustments  are  to  be  made,  and  whether  to  fit  a straight 
line  in  preference  to  other  types  of  curves.  However,  the 
location  of  a line  for  a given  series  of  data  is  determined 
because  only  one  line  satisfies  the  above  least  squares 
cri teri on . 

Another  way  to  determine  the  trend  of  previous  sales  is 
the  method  of  moving  averages.  Moving  averages  serves  to 
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eliminate  some  of  the  variability  not  attributable  to  the 
trend  in  an  attempt  at  its  identification.  (See  Figure  3). 

One  method  of  identifying  the  appropriate  trend  curve  is  to 
look  at  certain  characteristics  of  a curve's  moving  averages. 
The  reader  is  referred  to  the  book.  Mathematical  Trend  Curves 
by  Gregg,  Hossell,  and  Richardson  for  further  discussion  and 
examples  of  the  process  of  fitting  trend  curves  to  time 
series  data. 

A third  method  of  determining  trend  lines  is  that  of 
exponential  smoothing.  The  chief  advantage  of  exponential 
smoothing  is  that  only  a minimal  amount  of  storage  in  the 
computer  is  required;  only  five  computer  instructions  are 
necessary  and  can  be  used  repeatedly.  For  each  degree  of 
smoothing,  the  data  to  be  stored  are  quite  brief;  the  computa- 
tions are  quick  and  simple,  and  the  smoothing  constant  can  be 
adjusted  at  will  as  current  information  indicates  a need  for 
change.  This  method  has  been  widely  applied  in  simulation 
studies. 

Once  the  trend  line  has  been  determined,  it  is  necessary 
to  make  inferences  of  future  demand.  Having  obtained  the 
quation  of  a trend,  it  can  be  used  to  extrapolate  by  means  of 
estimating  a value  which  lies  beyond  the  range  of  values  on 
the  basis  of  which  an  equation  was  originally  obtained,  and 
thus  forecast  future  sales. 

Extrapolating  from  trends  is  a necessary,  though  highly 
speculative,  procedure  whose  success  depends  on  many  factors. 
The  basic  question  is  whether  the  forces  which  have  operated 
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in  the  past  will  continue  to  operate  in  precisely  the  same 
manner  in  the  future. 

b.  Estimation  of  Seasonality.  Until  now  the  emphasis 
has  been  placed  on  trend  estimations;  however,  from  a planning 
point  of  view,  estimation  of  seasonality  may  be  just  as  impor- 
tant in  forecasting.  For  example,  it  is  often  necessary  to 
take  into  consideration  fluctuations  other  than  trend  in 
production  smoothing  and  inventory  control.  Figure  4 depicts 
the  seasonality  in  sales  dollars.  Figure  5 illustrates  the 
use  of  a seasonal  index. 

There  are  two  types  of  seasonal  variations:  Those  whose 

pattern  remains  stable  over  a number  of  years,  and  those  whose 
pattern  changes  gradually  over  a period  of  time.  The  measure- 
ment of  constant  seasonals  involves  rather  complicated  theo- 
retical and  practical  considerations, 

c.  Types  of  Day.  There  are  many  factors  which  affect 
the  number  of  orders  received  each  day--for  example,  type  of 
day,  seasonality,  trend,  as  well  as  the  random  quality  of  the 
orders.  The  analyst  should  identify  the  type  of  day  because 
some  industries  may  have  a heavier  demand  on  Tuesday  than  any 
other  weekday.  Some  industries  may  have  one  type  of  day 
throughout  the  year,  while  some  may  have  as  many  as  10  or  more, 
To  illustrate  this.  Table  4 shows  the  classification  of  five 
types  of  days  of  customer  arrivals  in  a bank.  Once  the  type 

of  day  has  been  selected,  the  analyst  should  determine  the 
order  arrival  distrubution  which  is  to  be  discussed  in  the 
next  section  in  estimation  of  parameters. 
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Table  3 


Types  of  Day  of  Customers  Arrival 


Type 

1 

The  third  Friday  of  the  month 

Type 

2 

All  other  Fridays 

Type 

3 

The  first  day  of  the  month  if  it  is  a 
Tuesday,  Wednesday  or  Thursday 

Type 

4 

All  Mondays  and  the  first  two  Tuesdays  and 
Thursdays  of  the  month 

Type 

5 

Wednesdays,  and  the  last  two  or  three 
Tuesdays  and  Thursdays  of  the  month 

3. 

Estimati on 

of  Parameters 

The  discussion  which  follows  attempts  to  point  out  some  use- 
ful methods  for  estimating  system  parameters.  This  treatment  by 
no  means  exhausts  all  available  methods  for  parameter  estimation. 
The  reader  should  consult  a text  on  mathematical  statistics  or 
statistical  inference  for  a more  sophisticated  treatement. 

Three  steps  are  involved  in  defining  the  distribution  of  a 
random  variable.  First,  one  must  estimate  the  general  form  of 
the  distribution.  The  second  step  is  to  estimate  the  parameters 
of  the  hypothesized  distribution.  The  last  step  is  to  determine 
whether  or  not  the  hypothesized  distribution  adequately  repre- 
sents the  random  variable  in  question.  If  one  concludes  that 
the  hypothesized  distribution  in  unreliable,  the  analyst  should 
repeat  this  procedure,  starting  with  a search  for  a new  distribu- 


a.  Identifying  the  Distribution.  With  very  few  exceptions 
one  cannot  make  a reasonable  guess  regarding  the  distribution  of 
a random  variable  until  data  has  been  collected  which  can  be  used 
as  a guide.  Plotting  the  relative  frequency  distribution  of  the 
random  variable  under  study  is  often  helpful  in  determining  its 
probability  mass  function  or  density  function.  For  instance, 
from  an  order  record  of  203  days  one  can  construct  a frequency 
and  probaiblity  distribution  of  the  orders  arrival  such  as  in 
Table  4.  Figure  6 is  transformed  from  the  Table  4 probability 
distribution  into  a histogram  distribution,  which  may  suggest 
that  the  distribution  follows  that  of  a Poisson  distribution. 

Also  shown  in  Figure  7,  order  size  may  suggest  a negative  exponen- 
tial distribution. 

b.  Estimation  of  Distribution  Parameters.  Once  the  analyst 
has  identified  one  or  more  distribution  classes  which  he  feels 
adequately  represents  the  variable  he  is  studying,  he  then 
determines  the  numerical  values  of  the  distribution  parameters 

in  order  to  reduce  the  distribution  class  to  a specific  distribu- 
tion. When  the  hypothesized  distribution  is  a function  of  two 
parameters,  he  can  usually  estimate  these  parameters  from  the 
sample  mean  and  sample  variance.  Table  5 presents  common 
statistical  distributions  and  their  repsective  parameters,  and 
general  form  which  may  aide  the  analyst  in  determining  the 
probability  distribution  and  estimating  the  associated  parameters. 

c . Validating  and  Testing  the  Probability  Distribution. 
Having  hypothesized  that  a random  variable  is  characterized  by 
a specific  probability  distribution,  one  is  left  with  the  task 
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Table  4 
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Probability  Distribution  of  Orders  Arrival 


Orders 

Arrival 

Frequency 

Probability 

Cumulative 

Probability 

14 

19 

0-094 

0-094 

15 

42 

0-207 

0-301 

16 

44 

0-217 

0-518 

17 

41 

0-202 

0-720 

18 

26 

’Oa28 

0-848 

19 

11 

0-056 

0-904 

20 

9 

0-042 

0-946 

21 

4 

0-020 

0-966 

22 

3 

0-015 

0-981 

23 

2 

0-010 

0-991 

24 

1 

0-005 

0-996 

25 

1 

0-004 

1-000 

Total 

203 

1-000 

Figure  6 

Relative  Frequency  of  Orders  Arrival 
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Table  5 


Table  of  Common  Statistical  Distribution 


of  determining  whether  this  hypothesis,  which  at  best  is  usually 
nothing  more  than  an  educated  guess,  is  valid.  To  accomplish 
goodness-of-fi t , tests  (see  Table  6)  can  be  made  to  determine 
whether  sets  of  data  representing  the  outcomes  of  random  vari- 
ables are  adequately  characterized  by  known  probability  functions. 
4.  Determining  Relationships  Between  Components  of  the  System 
It  is  possible  to  represent  certain  relationships  between 
system  variables  with  other  known  variables.  It  may  be  properly 
represented  with  simple  or  multiple  linear  regression  equations. 
For  example,  regression  analysis  has  been  used  as  a statistical 
technique  for  estimating  costs.  Shipping  costs  may  be  ex- 
pressed as  a function  of  volume  shipped,  distance  traveled,  mode 
of  transportation  and  so  forth. 

Table  7 lists  the  data  of  44  freight  bills  and  related 
information.  The  variables  for  computer  step-wise  regression 
analysis  are  as  follows: 

X i - Shipping  origin  (zip  codes) 

X 2 - Shipping  destination  (zip  codes) 

X3  - Miles  shipped 
X4  - Weight  shipped 
X 5 - Freight  cost 

The  multiple  regression  equation  was  found  to  be: 

X5  = -55.2844  + 0.4943  X3  + 0.0079  X 4 
with  a multiple  correlation  coefficient  of  0.8983.  Bivariate 
frequency  distribution  between  dependent  variable  and  inde- 
pendent variables  is  shown  in  Figure  8. 
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Table  6 


COMMONLY  USED  STATISTICAL  TECHNIQUES  FOR  TEST  OF  GOODNESS-OF-FIT 


Techniques  Applications 


Analysis  of  Variance  Test  the  hypothesis  that  the  mean  (or 

variance)  is  equal  to  the  mean  (or 
variance)  of  the  corresponding  observed 
series 


Chi-Square  Test  Test  the  hypothesis  that  the  set  of 

generated  data  has  the  same  frequency 
distribution  as  a set  of  observed 
historical  data 


Kolmogorov-Smirnov  Test  This  test  may  be  used  as  an  alternative 

to  a chi-square  test.  It  has  two  advan- 
tages over  chi-square: 

1.  It  is  more  powerful  than  chi-square 
test. 

2.  It  can  be  applied  to  a very  small 
sample . 


Regression  Analysis  Regress  actual  series  on  the  generated 

series  and  test  the  resulting  regression 
equations 


Spectral  Analysis  The  analysis  provides  a means  of 

objectively  comparing  time  series 
generated  by  a computer  model  with 
observed  time  series 


Nonparametric  Tests  There  are  many  other  nonparametric  tests 

besides  kolmogorov-Smirnov  test  are 
available  for  test  of  goodness -of -fit . 

For  reference,  see  Siegel's  Nonparametric 


Statistics . 
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Summary 


The  process  of  data  base  construction  may  include  sampling, 
parameter  estimation,  forecasting,  regression  analysis,  and 
other  statistical  techniques.  Although  the  analyst  may  try  to 
extract  accurate  and  precise  information  at  each  step,  there 
always  exist  some  deviations  betwene  historical  data  and  the 
generated  data  due  to  chance  variation,  approximation,  and 
errors.  Validation  is  the  process  of  building  an  acceptable 
level  of  confidence  that  an  inference  about  generated  data  is 
correct  or  valid  for  the  actual  process.  Therefore,  valida- 
tion plays  a significant  role  in  the  data  base  construction. 

Validation  is  a process  of  acquiring  confidence  in  the 

>1 

generated  data  from  the  measurement  of  probability  because  a 
data  base  can  be  generated  that  exactly  matches  the  historical 
data,  but  there  can  be  no  assurance  the  data  generated  for  the 
future  will  match  the  future  events.  The  rules  for  validating 
and  generating  the  data,  therefore  are  sampling  rules  stemming 
entirely  from  probability  theory.  As  one  may  recall,  data 
construction  is  primarily  based  on  the  probability  distribu- 
tion which  was  derived  from  the  historical  data.  The  validity 
of  the  model  is  thus  made  probable  by  the  inductive  inference 
of  the  probability  theory. 
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